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The  purpose  nl  tins  ellorl  was  In  determine  the  research  areas  that  ap|ieat  most  critical  to  achieving 
man-computer  symbiosis  first.  an  operational  definition  ol  man-computer  symbiosis  was  developed  hv  (a) 
reviewing  and  summan/mg  whal  others  have  said  about  it.  and  (hi  attempting  to  distinguish  il  from  olhei  types  o| 
man-computer  relationships.  Most  general  views  ot  it  are  typified  hv  amplifications  ol  the  points  origmallv  laised  by 
Ijckhder  upon  introduction  ol  the  concept  in  I ‘>fiO  An  operational  definition  was  derived  hv  casling  (lie  various 
lyps-s.'.'l  man-computer  relationships  into  a systems  framework,  and  determining  the  ma|oi  distinguishing  leatuies  ot 
eacTi**ITie  major  distinguishing  tealurc  ol  man-computer  symbiosis  is  the  capability  ol  both  man  and  computet  to 
transceive  relevant  unrequested  information  fc>. 
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Item  20 (Continued)  - 
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Ring  the  (derived  definition.  basic  key  requirements  fit  j symbiotic  svsi.i  v.  .-i.  m i ■ > 1 these  included  li  I 

transceivers  ami  memories.  Op  graphics  techniques.  'f'  J langiia:'.  . ami  ijlC\  an  alien  In  mi  a 1 1 in.’  . upuhilily  Hies. 

basic  requirements  were  used  to  outline  arid  provide  structure  Mu  literature  miiv.-v^i.-  let.  the  state  nt  (In  art 

and  identity  critical  research  aieas 

l he  survey  Revealed  l hat  taster  scan  cathode  ray  tubes  hold  the  eie.i  test  pi  umi.  toiiis.insviubioit.  ■.  i 
hut  that  a critical  problem  accompanies  their  use  in  that  scan  conversion  methods  ate  leqirtrcd  I xisimp  m lliods  aie 
either  too  lime  consuming  or  too  memory  consunung^and  attempts  lo  improve  on  one  ol  these  aspects  hav.  all 
resulted  in  trader >ffs  on  the  othei  \ promising  technique  was  found  lor  reducing  the  miinbei  "i  vectors  cmnpri 
scene  llus  technique  would  t nr  cl  its  greatest  uiilitc  in  landom  ..at  t ul  1 n t ac  inocid.  Ih  i,.  i li 
display  pilose  densities  approach  those  possible  with  taslci  scan  I i 1 iino|,.; 

was  found  that  tew  distinct  impiovements  have  been  made  in  ovei  a decade  o|  i . ai. 1 and  that  n stnel.  d.  vi.  • 

exists  which  is  optimal  for  all  basic  ty  pcs  ol  input  that  the  man  would  e spec te.llv  need  to  . mplov 

Despite  consider  able  evidence  as  to  its  need.  Itllle  work  has  been  peiloiiu.d  in  vlemlin  mai  . m. 
precision  and  recall  . apahilily  Die  only  notable  research  I mind  vc  as  piovision  ol  a Simula  led  slnut  i.u  n m.  a c lot 
application  in  automotive  design  work  I (toils  to  extend  coinpulei  memory  c.ip.iciiv  . c lei  aiound  th 
development  ol  mtpioved  data  structures,  but  there  is  some  evidence  that  these  etlorts  aie  misdne.  led. 

1 here  are  no  methods  with  immediate  utility  lor  symbiotic  systems  lot  efficiently  lemoviu  luddei  In  . a 
surtacc’s  to  cited  a 'I)  cl  is  ploy  Most  methods  ate  loo  sloes  ccnii  attempts  at  impiov.  ill  ..nly  lesultm 

speed/memory  tradeoffs  Tlic  only  real  time  solution  is  unsuitable  lor  use  with  environment'  when  many  - i i 

move  independently . and  is  accomplished  only  at  the  expense  of  enormous  pro.  ssinv  power  dedicated  exclusivelv 
I to  hidden  Une/siirlace  removal 

> finally,  it  was  determined  that  the  most  piomising  solution  to  the  language  problem  lies  in  the  development  ol 
hyhnd  comnuimcation  techniques  using  a combination  ot  graphic  languages  and  small  scale  voice  input  methods 
Dus  could  o fl set  some  ol  the  disadvantages  ot  using  graphic  languages  alone  as  well  as  provide  a highly  efficient 
attention-getting  capability  and  enhance  the  rapport  man  feels  with  the  computer 

Flic  critical  research  areas  that  were  identified  were  summarized  in  table  form  In  the  author's  opinion 
advancements  in  these  areas  would  effect  the  most  rapid  closure  on  achieving  the  type  ot  system  required  tor 
man-computer  sy  mbiosis 
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Problem 

Wiih  the  increasing  conplcxitv  of  modern  weapon  s\ stems  and  simulators,  new  methods  are  required 
lor  providing  elTicient  and  elleetiv#  interlace  between  the  instructor  and  the  computer  in  (light  training 
devices.  Over  the  scats,  in  efforts  quite  unrelated  to  training  simulation,  various  studies  have  been 
conducted  in  attempts  to  create  unique  man-computer  partnerships  for  solving  problems.  When,  in  such  a 
partnership,  both  man  and  computer  complement  and  benefit  each  other,  the  relationship  is  termed 
s>  mbiolic  I he  problem  was  to  find  out  what  research  areas  appear  most  critical  to  achieving  man-computer 
s\  tnhiosis  lor  potential  future  applications  in  the  design  ol  simulator  instructor, 'operator  stations. 

Vpproach 

Ihe  approach  was  to  determine  a precise  operational  definition  of  man-computer  sy  mbiosis  and  use  it 
to  identity  the  requirements  of  a symbiotic  system.  The  requirements  were  then  used  to  outline  and 
provide  structure  to  a literature  survey  to  determine  the  state-ot-ihc-art  and  identify  critical  research  areas. 

Results 

An  operational  definition  of  mail-computer  symbiosis  was  derived.  Ihe  major  teat  lire*  that 
distinguishes  it  from  other  types  of  man-computer  relationships  is  the  capability  of  both  partners  to 
transceive  relevant,  unrequested  information.  Key  requirements  lor  a symbiotic  system  are  transceivers  and 
mcinoiies.  graphics  techniques,  a language,  and  an  attention-getting  capability  \ survey  ol  literature  in 
these  four  specific  areas  was  conducted  and  critical  research  areas  were  identified 

('(inclusions 

Improved  scan  conversion  methods  for  raster  scan  cathode  ray  tubes  ate  required  lor  widespread 
development  and  use  ol  symbiotic  systems  Also,  some  technique  is  needed  lot  providing  a simulated 
short-tcim  memoiy  for  man's  use  in  an  interactive  graphics  environment.  Hidden  line  and  hidden  surface 
displav  techniques  are  presently  slow  and  inefficient  Advances  m this  area  would  constitute  a breakthrough 
in  display  technology.  Voice  communication  with  the  computer  is  probably  the  most  influential  change  we 
wall  observe  in  the  neai  future  that  will  greatly  enhance  man-computer  symbiosis  However  improvements 
in  word  and  phrase  recognition  rehabihtv  are  needed.  I inally.  visual  resolution  processing  techniques, 
which  icducc  display  complexity  by  taking  advantage  ol  human  peiccption  characteristics,  appears  to  be  .t 
promising  compromise  between  limiting  the  information  to  be  displayed  and  unrealistically  increasing  the 
computer  resources  icquiicd  lor  the  display  portion  ol  the  sy  stem. 
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This  report  was  written  in  partial  fulfillment  of  course  requirements  lot  the  M.S. 
degree  in  Compute!  and  Information  Science  at  the  Ohio  Stale  University.  Columbus. 
Ohio.  The  author  sincerely  thanks  Professor  K.  I Trnst  for  Ins  helpful  suggestions,  as 
well  as  recommendation  and  personal  loan  of  a number  of  papers  used  in  the  course  ol 
the  survey.  In  addition,  thanks  go  to  Mr.  J I).  Basinger,  previously  with  the  Air  force 
Human  Resources  Laboratory.  Advanced  Systems  Division.  Wright  Patterson  Air  force 
Base,  Ohio,  for  providing  personal  notes  and  papers  on  recently  developed  computer 
image  generation  techniques. 

The  work  reported  and  results  obtained  arc  of  direct  use  in  helping  to  satisfy  the 
objectives  of  project  61  14,  Simulation  Techniques  for  Aerospace  Crew  Training,  and  task 
61  1420.  Advanced  Training,  with  the  author  serving  as  both  Project  and  Cask  Scientist 
Therefore,  although  nearly  two  years  have  passed  since  the  work  was  originally  performed 
at  the  University,  it  was  believed  to  be  in  the  best  interests  ol  the  Air  force  to  publish  the 
results  at  this  time  for  distribution  to  the  scientific  community. 


' 'X 


I  Mil  I <>!  CON  1 1 MS 


Page 

I  Introduction  

II.  Approach  ami  Overview  5 

III  Results ■ 

An  Operational  Dclintiion  of  Man-Computer  Symbiosis  5 

key  System  Requirements 

Cieneral  Survey  and  Iclcnti I ication  of  Critical  Problems l> 

IV.  Overview  and  Conclusions  32 

Status  Transceivers  and  Memories 32 

(■rapines  Techniques 33 

Language 33 

Summary  of  Critical  Research  Areas  ....  33 

References 35 

I 1ST  01  ILLUSTRATIONS 

figure  Page 

1 Man-computer  interactive  system  diagram 

2 I '..x tension  ol  man  system  N 

3 Partner  of  man  system  X 

4 Symbiotic  system  X 

5 Key  requirements  for  a symbiotic  system  10. 

(>  Arrangement  of  three  types  of  tiles  in  a graphics  system  IP 

7 Summary  of  key  results  of  Barrett  and  Jordan  experiment  21 

X Results  of  Bussell  and  Taxin  experiment  22 

9 Simplified  illustration  of  Schumacker's  cluster  prioritizing  technique  26 

I IS  I 01  TABU  S 

table  Page 

1 Distinguishing  Characteristics  of  Three  Types  of  Man-Computer  Systems  . . l) 

2 Results  of  Comparison  of  Warnock  and  Bouknight  Methods  25 

3 Summary  of  Critical  Research  Areas  34 


MAN-COMI'UTf  R SYMBIOSIS  T1IKOUC.II 
INTI  RACTIVI  (iRAI’IIK'S  A SURVI Y AND 
IDI  NTIf  R ATION  Ol  CRIIK  Al  RI  SI  ARCH  ARI  AS 


I.  INTRODUCTION 

The  purpose  of  this  paper  is  to  determine  and  survey  the  research  areas  that  appeal  most  critical  to 
achieving  man-computer  symbiosis.  Since  1900.  when  Licklider  published  Ins  now  classic  and  often 
referenced  paper  introducing  the  term  ‘man-computer  symbiosis’  (I  icklider.  19601.  a tremendous  volume 
of  work  has  been  performed  in  interactive  computer  graphics.  The  sunk  is  diversified,  covering  both 
hardware  and  software  research  and  ranging  from  developments  in  graphic  input  devices  and  general  display 
technology  to  graphic  data  structures,  programming  languages,  and  soltware/mathematical  tcchnu|iies  loi 
retrieving,  sorting,  and  manipulating  graphic  data.  Applications  of  interactive  graphics  are  unbelievable  in 
number  and  range  from  circuit  and  architectural  design  to  complex  problem  solving. 

Because  man-computer  symbiosis  appears  to  be  one  of  the  major  goals  ol  and  inspirations  lor  much 
of  this  work,  and  because  its  goals  are  essentially  to  provide  the  ultimate  in  man-computer  partnership,  it 
was  believed  a worthwhile  effort  to  find  out  where  we  stand  in  its  regard.  Six  years  after  licklider ’s  paper. 
Sutherland  ( 1966)  identified  several  outstanding  problems  that  remained  unsolved  at  that  time,  four  years 
later,  in  1 ‘>70,  DeCireene  ( 1970)  surmised  that  we  were  still  many  years  away  from  operable  man-computer 
symbiotic  systems.  And  now  what  is  the  status  of  affairs?  More  important,  what  are  the  areas  in  which  we 
can  channel  our  best  efforts  to  bridge  the  major  gaps  that  remain'1  finally,  what  is  the  state  ol  the  art  in 
those  areas  of  interactive  graphics  that,  if  mastered,  can  effect  the  most  rapid  closure  on  the  goals  ol 
man -computer  symbiosis?  This  paper  is  an  attempt  to  answer  these  kinds  ol  questions. 


II.  APPROACH  AND  OVI  K VIIAV 

A survey  of  work  in  critical  problem  areas,  which  is  one  of  the  purposes  here,  requires  first  the 
recognition  of  critical  problems.  This  in  turn  requires  some  feeling  lor  the  key  requirements  ol  a 
man-computer  symbiotic  system  to  which  any  critical  problems  would  expectedly  be  related.  To  ascertain 
system  requirements  necessitates,  in  turn,  a definition  ol  the  system  itself  I he  approach,  then,  wxs  pretty 
much  dictated  by  the  purpose  of  the  work  and  included  the  following  steps: 

1 Determine  tin  operational  definition  of  man-comp  liter  symbiosis  This  proved  to  be  no  small  task 

I very  one  intuitively  “knows"  what  man-computer  symbiosis  is;  therefore,  no  one  has  apparently  felt  a 
need  to  define  it  from  an  operational  standpoint.  The  approach  here  was  to  review  and  absorb  what  others 
have  said  about  it.  attempt  to  contrast  it  with  other  types  of  man-computer  relationships;  and.  finally, 
define  it  from  a systems  viewpoint. 

2 I’roject  key  system  requirements.  Using  the  derived  definition  ol  man-computer  symbiosis,  three 
general  requirements  of  a symbiotic  system  were  projected.  These  requirements  were  stated  in  terms  ot 
“what  basically  do  we  need  in  such  a system." 

3.  Perform  a general  survey  and  identify  critical  problems.  A general  survey  ol  areas  related  to  the 
established  system  requirements  was  conducted.  Tins  provided  an  excellent  overview  ol  the  basic  state  ot 
the  art  and  revealed  what  are  believed  to  be  live  major  critical  problem  areas.  Advancements  in  these  areas 
would,  in  the  author’s  opinion,  most  greatly  benefit  the  progress  toward  man-computer  symbiosis. 


III.  Rl  SI  LTS 

An  Operational  Definition  of  Man-Computer  Symbiosis 

General  Viewpoints  licklider  (I960)  originally  discussed  man-computer  symbiosis  in  terms  ol 
combining  the  mutually  beneficial  talents  of  man  and  computer  by  means  of  (a)  minimi/ing  their 
incompatibilities  as  potential  partners,  and  (b)  allocating  tasks  to  them  in  accordance  with  their  respective 
talents  lie  cites  their  major  points  of  incompatibility  as  speed  and  language  mismatches  and  discusses  at 
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some  length  their  comparative  capabilities.  A few  examples  of  the  latter  are.  (a)  men  are  noisy,  narrow 
band,  error  prone  devices  with  many  parallel  channels;  computers  are  fast  and  accurate  but  do  only  one 
thing  at  a time;  and  (h)  men  arc  flexible  and  can  program  themselves  contingently  on  the  basis  of  new 
information;  computers  are  single-minded  and  constrained  by  their  preprogramming. 

lo  varying  extents,  most  references  to  man-computer  symbiosis  as  well  as  interactive  graphics  per  se 
essentially  amplify  on  these  basic  concepts.  David  (l%7)  discusses  at  length  the  differences  between  man 
and  computer  and  concludes  generally  that  an  effective  interface  must  perform  recoding  and  buffering.  He 
also  points  out  some  ol  the  considerations  which  limit  the  extent  to  which  man  can  be  made  to  adapt  to 
the  computer. 

Miller  ( I d(>5 ).  emphasizing  mainly  the  psychological  and  physiological  capabilities  and  limitations  of 
man,  states  that  the  man-computer  relationship  must  be  synergistic  (mutually  reinforcing)  as  well  as 
symbiotic  (mutually  beneficial),  lie  identifies  three  central  human  activities  in  problem  solving  (browsing, 
hypothesis  formation,  and  hypothesis  testing),  and  relates  to  each  some  implied  requirements  of  a 
man-machine  problem  solving  sy  stem.  I or  example,  browsing  necessitates  individualized  selective  display  of 
timely  information;  and  hypothesis  formation  requires  a display  which  fosters  the  conceptualization  of 
appropriate  actions. 

Smith  ( I ‘>70),  in  describing  the  design  features  of  an  interactive  graphics  system  to  solve  numerical 
problems,  cites  the  following  basic  system  requirements;  displays  must  be;  (a)  self-explanatory,  and  (b) 
self-helping  (i.e.,  must  check  user  inputs);  user  actions  that  are  required  should  be  simple  and  obvious,  the 
system  sliou Id  possess  optional  verbosity  so  it  can  be  used  effectively  and  comfortably  by  a novice  or 
experienced  user;  and  feedback  should  be  provided  to  the  man  when  he  acts.  All  of  these  requirements  will 
be  recognized  as  one  form  or  another  of  accommodating  the  human’s  possession  or  lack  of  capability  or  his 
preference. 

Both  Matsa  (1%‘>)  and  Newman  and  Rogers  (l‘)66)  highlight  optimal  task  allocation  as  being  the 
requisite  lor  an  interactive  graphics  system.  Matsa  states,  for  example,  that  regardless  of  the  application, 
there  is  a certain  amount  ol  both  man  and  computer  required;  and  that  interactive  graphics  involves 
combining  the  best  of  the  capabilities  of  each  by  allocating  tasks  accordingly  . Somewhat  in  contrast  to  tins 
and  some  years  earlier.  Licklidcr  ( I l)D2 ) pointed  out,  a few  years  subsequent  to  his  introduction  of 
nian-cornputcr  symbiosis,  that  a strict  task  allocation  is  not  really  feasible  because  we  cannot  |ttsl  divide  a 
task  into  par ts.  Ills  view  is  that  partitioning  a task  into:  (a)  fine  detail  (computer’s  job),  and  ( b ) the  overall 
(man's  job)  is  not  feasible  because  details,  by  themselves,  are  meaningless,  and  wholes  do  not  exist  without 
parts.  While  vague  from  the  standpoint  of  suggesting  a potential  solution,  Licklider’s  article  conveys  the 
message  ol  "whole  > sum  of  parts’  and  provides  a key  concept  in  man-computer  symbiosis. 

In  summary,  both  symbiosis  and  interactive  graphics  in  general  are  typically  viewed  as  problems 
involving:  (a)  minimizing  differences  between  man  and  computer,  mainly  through  appropriate  interface; 
and  (b)  optimally  allocating  tasks  to  man  and  computer  based  on  their  respective  talents.  Distinctions 
between  symbiotic  systems  and  general  interactive  graphics  systems  arc  rarely,  it  ever,  made  in  the 
literature.  A key  point  made  by  Licklidcr  subsequent  to  his  introduction  of  man-computer  symbiosis 
implies  that  straightforward  task  allocation  does  not  necessarily  result  in  a symbiotic  system. 

,-t  Systems  yiewimint.  file  basic  idea  for  the  operational  definition  of  man-computer  symbiosis 
developed  herein  came  hum  an  article  by  Jacks  (l%8).  In  describing  and  rationalizing  the  design  ol  an 
interactive  system  used  at  Oeneral  Motors  to  solve  automotive  design  problems.  Jacks  presents  the  hum  an 
and  computer  as  two  systems,  each  possessing  receivers,  a processor  and  memory,  and  transmitters  (see 
I igure  I).  I his  viewpoint,  ol  course,  is  the  same  as  that  adopted  in  control  system  design  and  analysis 
(Bekey.  I ‘>70)  Its  use  by  lacks  in  the  specific  context  of  interactive  graphics  suggests  a useful  means  of 
distinguishing  sy  mbiotic  systems  limn  those  which  are  noil-symbiotic. 

Jacks  discusses  the  duel  characteristics  of  each  component  of  the  human  and  computer  sy  stems,  lie 
remarks  that  the  central  reason  for  using  visual  displays  in  interactive  systems  is  man’s  ability  to  accept 
visual  data  (as  opposed  to  oilier  types  ol  data)  at  a high  rate.  Considering  the  processor  section  of  man.  he 
remarks  that  little  is  known;  however,  man  appears  to  operate  in  a burst  mode  he  accepts  data  rapidly, 
mulls  over  information  to  decide  on  an  act.  then  releases  a burst  of  information  to  the  computer.  On  the 
computer  side  Jacks  discusses  ways  to  accomplish  matching  I/O  to  the  man  (here  again  minimizing  die 
dilteiences  between  man  and  computer).  I Its  suggestions  are  both  unique  and  valuable  and  seem  to  spring 
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Figure  I.  Man-computer  interactive  system  diagram. 


from  his  systems  view  of  the  situation.  (One  of  his  suggestions  in  particular,  which  will  be  discussed  later  m 
this  paper,  is  to  include  on  the  computer  side  a simulated  short-term  memory  for  the  man.) 

By  applying  the  systems  viewpoint  as  Jacks  did.  various  types  ol  man-computer  relationships  can  be 
categorized.  It  is  proposed  that  there  are  three  basic  types  of  relationships:  (a)  the  computer  as  an 
extension  of  man ; ( b)  the  computer  as  a partner  of  man,  where  the  respective  tasks  ol  each  are  assigned  via 
straightforward  task  allocation:  and  (c)  man-computer  symbiosis,  figures  2.  3.  and  4 depict  system  block 
diagrams  for  each  of  these  types  of  relationships. 

In  the  extended  man  system  ( I igure  2).  man  receives  external  inputs  about  the  task  to  be  performed 
and  his  major  interaction  with  the  computer  is  to  request  a job  to  be  done  and  receive  a reply  (data).  Ibis  is 
a strict  master/slavc  relationship  wherein  the  computer  receives  no  inputs  except  Irom  the  human  and  acts 
(responds)  only  when  requested  to  do  so.  In  the  partner  ol  man  system  (figure  31  both  computer  and  man 
receive  certain  external  inputs  relative  to  their  respective  tasks.  Man  acts  as  an  overseer,  in  addition  to 
performing  his  own  tasks.  His  major  interaction  with  the  computer  is  to  request  and  receive  status 
information  and,  based  on  that  information,  override  the  computer’s  performance  of  its  own  tasks  as 
required.  In  t fie  symbiotic  system  (f  igure  4),  both  man  and  computer  can  request  information  Irom  each 
other  as  well  as  (a)  respond  to  a request  for  information,  and  (b)  provide  unrequested  information 

It  is  proposed  that  the  primary  distinguishing  features  ol  the  three  systems  are  (a)  the  major  ty  pes  ol 
information  traded  between  man  and  computer  and.  equally  important,  and  (h)  whether  a transmittal  to 
the  man  on  the  part  of  the  computer  is  or  is  not  always  performed  solely  in  response  to  a request  trout  the 
man.  Table  I summarizes  tire  three  systems  in  terms  id  these  distinguishing  characteristics.  Ol  particular 
note  is  that  it  is  only  in  the  symbiotic  system  that  the  computer  independently  generates  information  (oi  a 
request  for  information) 

Tins  proposed  man-machine  system  classification  scheme  appears  to  be  in  accord  with  the  views 
brought  out  originally  by  Ucklider  (l%0)  regarding  man-computer  symbiosis.  Most  important  is  that  it 
incorporates  fi is  ‘whole  >sum  of  parts'  concept  ( l acklider.  I‘)(i2)  by  demanding  that  in  a symbiotic  system, 
the  computer  must  be  capable  of  independently  acting,  in  a dynamic  sense,  on  the  basis  ol  the  overall 
system  goals  and  the  total  system  status  at  any  given  moment . Hi  is  includes  acting  on  the  basis  of  what  the 
man-component  of  the  system  is  doing  and  has  done.  In  contrast  to  its  role  in  the  other  systems,  the 
computer  in  a symbiotic  system  must  be  able  to  transmit  relevant  unrcifuestetl  information  relevant  to  the 
goals  of  the  system,  and  unrequested  in  the  sense  that  the  computer  independently  decides  it  is  appropnate 
and  of  potential  benefit  for  the  man  to  receive  it 
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\n  Operational  Ocjinilion.  Man-computei  s>mbii)sis  is  the  joining  of  two  suhsystems  ( nun  and 
computer)  into  a single  efficient  system  wherein  both  man  and  computet  have  the  illteicomnumication 
capability  of  transceiving  intorniation  that  is  relevant  to  the  goals  ol  the  system.  I he  inhumation  received 
hv  one  or  the  other  may  ot  may  not  have  been  requested.  The  decision  by  one  ol  the  subsystems  to 
transmit  information  is  dynamically  made  as  a function  of:  (a)  the  overall  system  goals.  ( hi  the  curient  and 
past  system  states,  and  (c)  the  current  and  past  activities  of  the  other  subsystem. 

kev  System  Requirements 

I tom  the  definition  in  the  previous  paragraph  and  the  major  distinguishing  features  liable  II  ot 
various  sy  stem  ty  pcs.  it  appears  that  the  key  requirement  in  a sy  mbiotic  system  is  the  capability  ol  man  and 
computer  to  interchange  relevant  un requested  information.  There  ate  a number  ol  system  implications  that 
can  be  derived  from  this  general  requirement. 

The  determination  ol  whether  or  not  information  is  relevant  implies  a need  to  know  the  status  ol  the 
total  system  relative  to  its  goals  Therefore,  both  man  and  computet  require  a s/u/trv  transceiver  tind 
nn /non.  The  fact  that  the  information  to  be  transmitted  is  both  relevant  and  unrequested  implies  a 
required  capability  lot  arousing  the  attention  of  (lie  receiving  subsystem,  immediately  it  required,  lints,  an 
attention-getting  ea/>ahilit\  is  needed.  Since  the  data  to  be  transmitted  is  unrequesled  (unanticipated)  and 
constitutes  information,  it  is  imperative  that  it  be  formatted  for  presentation  in  a way  that  will  be  clearly 
understood  by  the  receiver.  Therefore,  graphics  Icchnupics  are  required  for  manipulating  and  tormaUing 
information  to  be  displayed  finally,  the  fact  alone  that  the  data  to  be  transmitted  constitutes  inhumation 
means  that  it  inherently  contains  new  knowledge  for  the  receiver  that  is  relevant  to  system  peitormance. 
Therefore  an  unrest rictive.  flexible,  and  easily  applied  means  is  required  to  "get  the  point  across  ."  Thus  we 
need  a language 

The  above  key  system  requirements  are  summarized  in  figure  5 in  terms  of  then  emanation  Irotn  the 
general  requirement  for  transceiving  relevant  unrequested  information  An  attention-getting  capability 
would  expectedly  be  realized  by  means  ol  the  status  transceiver  and/or  the  language  Therefore  it  is 
considered  a companion  requirement  to  one  or  both  ol  them  rather  than  an  independent  requirement 

In  summary,  then,  three  key  requirements  ol  a symbiotic  system  have  been  identified  (a)  status 
transceive  is  and  memories.  I b)  graphics  techniques,  and  Ida  language  I liese  requirements  emerge  ti.it  u tally 
Iroin  the  necessity  that  each  subsystem  (mail  and  computer)  be  able  to  tiansceivc  relevant  unrequested 
information  Next,  we  turn  to  a general  review  ol  existing  capabilities  in  areas  related  to  satisty  mg  each  ol 
these  three  key  requirements. 

(•eneral  Survev  and  Identification  of  Critical  Problems 

In  this  section,  a survey  is  presented  ol  existing  capabilities  relevant  to  satislymg  the  key 
requirements  lor  a symbiotic  system  The  purpose  ol  this  survey  was  to  acquire  a general  undeistanding  ot 
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the  state  of  the  art  and.  equally  important,  to  determine  and  explore  those  specific  areas  of  research  which 
seem  most  critical  to  the  development  of  man-computer  symbiotic  systems. 

Status  Transceivers  and  Memories.  hxisting  capabilities  to  satisfy  requirements  lor  status  transceivers 
and  memories  can  be  partitioned  into  three  groups:  displays,  input  devices,  and  memory  aids 

( I ) Displays 

Psychologists  estimate  that  as  much  as  80'  of  our  sensory  data  is  received  in  the  form  ot  visual 
stimuli  (Van  Dam.  lflbh).  This  fact,  plus  the  capability  of  man  to  accept  visual  information  at  high  rates 
(Jacks.  I‘>68).  makes  a computer  driven  display  an  effective  and  defcndahle  selection  as  a means  lor 
coniputer-to-man  communication.  At  the  same  tune,  however,  effective  visual  communication  using 
displays  is  not  unbeset  with  problems.  These  problems  arise  largely  out  of  the  need  to  accommodate  the 
perceptual  characteristics  of  the  human. 

Considerable  data  are  available  on  the  visual  perception  characteristics  ol  man  and  the  resulting 
implications  for  display  selection  and  design  (see  (Semple.  Heaps.  Conway.  A liu me* tie.  I’*7I  ) lor  an 
excellent  review  of  related  research  and  findings  and  extensive  list  ol  references).  Cicncrally  the  major 
factors  which  directly  affect  display  selection  and  engineering  are  luminance,  which  man  perceives  as 
brightness  according  to  a power  law:  luminance  contrast,  which  is  essentially  die  ratio  ol  signal  minus 
background  luminance  to  total  luminance,  including  that  contributed  by  reflected  ambient  lighting, 
resolution:  and  display  refresh  rate  (Could.  I lX>H ).  With  the  exception  ol  luminance  contrast,  these 
requirements  are  generally  capable  o)  being  satisfied  using  existing  display  technology.  With  the  further 
exception  of  display  refresh  rate,  the  requirements  are  in  practice  satisfied  in  most  existing  applications 
The  key  message  which  emerges  from  the  literature  (I  rerich  & leger.  D>72.  Noll  1 ‘>7 1 .Sutherland.  D>bb) 
is  that  display  refresh  rate  presents  a problem  not  uncommonly  encountered  m interactive  graphics  work 

(a)  Display  Refresh  Requirements 

Display  refresh  rate  refers  to  the  frequency  with  w ditch  the  visual  scene  on  a display  is  refreshed  oi 
rewritten.  Problems  related  to  it  arise  when  the  refresh  rate  is  close  to  the  critical  dicker  fusion  (ctf) 
frequency  of  the  individual.  This  is  the  frequency  at  and  below  which  flicker  in  the  visual  scene  is 
noticeable.  The  eft  is  dependent  upon  the  individual  as  well  as  several  other  factors,  including  display 
brightness,  contrast,  and  the  ratio  of  viewing  distance  to  display  diameter  (Gould.  1068.  Semple  ef  al .. 
1071)  While  flicker  may  he  controlled  and  display  refresh  rales  reduced  by  using  long  persistence 
phosphors,  this  limits  the  rate  at  which  the  display  can  be  updated.  Therefore,  the  refresh  rate  itsell  is  the 
principal  parameter  in  controlling  flicker  (Semple  et  al..  1071 , p.  405). 

A generally  adopted  rule  of  thumb  is  to  refresh  the  display  at  least  (5  to  40  times  pci  second  to  avoid 
flicker  (DcGreene.  1070).  It  may  appear  simple  enough  to  merely  adopt  this  rule  of  thumb  and  thus  resolve 
the  problem  without  further  ado.  However,  minimum  refresh  rates  are  often  required  in  high  density 
displays  because  there  is  an  inverse  relationship  between  the  refresh  rate  and  the  number  ot  picture 
elements  to  be  display ed  ( Semple  et  al..  1 07 1 . p.  4 1 I ).  This  in  turn  is  due  to  haiulw  idtli  limitations  (in  bits 
per  second)  on  the  computer  display  system  itself.  A typical  manifestation  of  the  problem  is  described  by 
I rcncli  and  leger  ( 1072)  who  cite  flicker  as  the  major  problem  encountered  in  development  of  a giaphical 
on-line  design  system.  Then  system  capacity  allowed  the  storage  and  display  of  2.000  vecfois.  howevei. 
flicker  occurred  at  800  Hie  principal  solution  iccoinniended  to  help  resolve  the  display  letrcsh  problem  as 
well  as  several  others  is  to  use  raster  scan  displays  ra the i than  the  mote  conventional  landom  scan  types 
(Gould.  1 0(>8:  Jordan  & Bairett.  1 074:  Opliii.  Rankowil/.  Shepheid.  A Spiniad.  1068) 

(b)  Raster  Scan  ( athode  Ray  lube  l(  R I ) Displays 

Raster  scan  is  one  ot  five  scanning  methods  used  in  computet  data  display  s ( Davis.  I'h>>),  p nol  It  is 
accomplished  by  deflecting  the  electron  beam  repetitively  ou  i a airs  ni  hon/nnlal  lines  on  file  viewin'.' 
screen  and  intensifying  portions  of  (lie  lines  in  response  to  input  commands.  I Inis,  inhumation  al  position 
x.y  on  the  screen  is  updated  each  time  the  electron  beam  scans  line  v ( ommeicial  IV  employs  mteilaced 
raster  scanning,  and  it  is  with  this  technique  that  refresh  tales  mav  be  tcdticed  Interlaced  scanning  icteis  to 
the  technique  of  scanning  (and  updating  information  on)  alternate  lines  on  each  of  two  successive  passes 
down  the  screen  li  e.,  lines  1 .4.5  etc.  on  pass  I and  lines  2.4.0  etc.  on  pass  2 ) 

The  reason  that  interlaced  scan  can  reduce  required  retiesh  rates  is  due  to  man  s |xusislencc  ol  vision 
(the  tendency  lor  a sensation  resulting  from  a visual  stimulus  to  remain  alter  removal  of  the  stimulus)  Ibis 


factor,  plus  flic  close  proximity  of  adjacent  scan  lines,  results  in  an  effective  refresh  rate  in  an  interlaced 
scan  system  that  is  lusher  than  the  actual  refresh  rate  of  each  screen  element.  Thus,  a TV  system  actually 
updates  each  displayed  element  at  a rate  ol  30  pet  second,  and  though  tins  is  below  “typical"  cits,  flicker  is 
not  apparent 

Several  other  advantages  to  raster  scan  displays  in  interactive  graphics  work  are  discussed  by  Jordan 
and  B.inott  t 1 ‘>74 > Chief  among  them,  and  of  great  relevance  to  widespread  development  and  use  of 
symbiotic  systems,  is  the  cost  lactor.  I or  example,  in  one  ol  the  earliest  uses  ol  raster  scan  displays  tor 
inleiaclive  graphics  (Oplui  el  at.,  I96X),  a network  ol  ten  display  consoles  was  developed  for  less  than  SXO 
thousand.  A similar  network  using  random  scan  character  and  vector  generated  displays  would  have  cost 
trout  S3 00  thousand  to  XMlil  thousand. 

Another  significant  advantage  is  that  obtaining  algorithms  to  remove  hidden  lines  and  surfaces  fomi  a 
picture  can  be  accomplished  with  little  extra  effort  using  a raster  display  . This  is  because  the  more  efficient 
algorithms  are  similar  to  those  recpiired  to  translate  the  standard  x-y  picture  description  into  a raster 
format . 

Die  major  disadvantage  of  raster  scan  displays  i:>  that  scan  conversion  techniques  are  required  to 
perforin  the  translation  from  x-y  to  raster  format  (Newman  & Sproull.  1973;  Noll.  1971).  Tins  can  be 
accomplished  using  scan  conversion  hardware,  but  the  equipment  is  fairly  complex,  adding  to  overall 
system  cost  and  reliability  problems.  Therefore  scan  conversion  is  normally  accomplished  using  software 
techniques. 'This,  of  course,  adds  to  the  computati  mal  load.  Several  scan  conversion  algorithms  are  available 
and  are  discussed  in  a subsequent  section  of  this  paper.  However,  it  will  be  seen  that  scan  conversion  is  a 
critical  problem  area. 

(c)  Random  Scan  CRT  Displays 

Despite  their  higher  initial  cost,  higher  refresh  rate  requirements,  and  the  associated  display  density 
limitations  discussed  previously,  random  scan  displays  have  been  used  far  more  widely  in  the  past  for 
computer  graphics  applications  than  have  raster  scan  types.  This  is  probably  due  to  (a)  the  lack  of  need  for 
scan  conversion,  (b)  excellent  hardware  character  and  vector  generation  capabilities  existent  in  random  scan 
displays,  and  (e)  high  commercial  availability  and  advertisement. 

However,  in  1971,  Noll  predicted  a distinct  trend  away  from  random  scan  displays  for  several 
noteworthy  reasons.  At  Bell  Telephone  laboratories,  Noll  and  associates  had.  for  some  years,  used  a simple 
dot  plotting  CRT  in  man-machine  communication  research.  The  reason  hardware  vector  and  character 
generators  were  not  used  was  because  of  their  expense  and  associated  reliability  problems  (they  were  a 
“continuing  source  of  trouble”  (Noll,  1971,  p.  143)).  The  dot  plotting  approach  was  not  satisfactory, 
however,  due  to  flicker  problems  encountered  with  high  data  density  (over  4,000  dots)  and  the 
considerable  core  storage  required  to  hold  each  x-y  coordinate  pair.  Therefore  a scanned  display  system  w as 
implemented  for  use  with  a significant  resulting  improvement  over  their  previous  system. 

Random  scan  CRT  displays  of  a wide  variety  are  commercially  available  m a highly  competitive 
market.  They  may  be  basically  classified  according  to  the  type  of  information  presented:  alphanumeric, 
graphic,  or  situation  displays  (Davis.  1969).  In  alphanumeric  displays,  alphanuinerics  and  some  symbols  are 
presented  in  page  format  on  the  CRT.  In  graphic  displays,  line  drawing  capability  is  added  to  an 
alphanumeric  capability.  Situation  displays  may  have  characteristics  of  both  alphanumeric  and  graphic- 
displays  in  addition  to  an  added  capability  of  presenting  some  form  of  background  data  (e.g..  radar  or 
sonar). 

The  most  common  type  used  in  interactive  graphics  work  is  the  graphic  type  CK  f.  I Tut  cost  is  large!', 
a function  of  the  hardware  options  available.  These  include  scaling,  zooming,  and  21)  and  3D  line  rotation, 
plus  variable  intensity  for  achieving  depth  cueing.  A review  of  some  specific  typical  graphics  displays  is 
provided  by  David  ( 1969).  Due  to  the  wide  competitive  market,  there  appears  to  be  no  inherent  problem  in 
obtaining  (for  a price)  a random  scan  CRT  display  suitable  for  a specific  application,  given  that  the 
application  can  live  with  the  basic  data  density  limitations  and  associated  relTesh  requirements  mentioned 
earlier. 

(d)  Other  Display  Types 

Other  CRT  devices  are  available  (e.g.,  fiat  panel  displays  and  direct  view  storage  tubes);  however,  lor 
various  technical  reasons,  none  have  achieved  the  flexibility,  relatively  low  cost,  and  wide  availability  ol  the 
conventional  raster  or  random  scan  devices.  Comprehensive  reviews  of  alternative  CRT  devices  are  provided 
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in  Davis  ( |9<>9).  Kleinian  (1969).  and  Newman  and  Sproull  (1973).  Ihe  direct  view  storage  tube,  lor 
example,  appears  at  first  glance  to  provide  a potential  answer  to  the  refresh  problem,  because  it  contains  a 
built-in  storage  grid  behind  the  front  screen.  However  selective  erasure  ot  screen  elements  is  difficult,  if  not 
impossible  (Newman  & Sproull.  1973).  and  picture  quality  is  poor  compared  with  that  achieved  by 
conventional  random  and  raster  scan  CRTs. 

Engineering  development  work  in  the  display  area  is  abundant,  as  might  he  expected  due  to  the 
potential  market  lor  improved  display  devices  (cl  II  1.1  C onference.  I ‘>7 2 1 However  it  is  probable  that 
these  other  display  devices  will  only  achieve  the  versatility,  low  cost,  and  wide  coinmerical  availability  ot 
the  more  conventional  CRT  types  a number  of  years  from  now  Thus  their  widespread  use  in 
man-computer  symbiotic  systems,  if  it  occurs  at  all.  lies  sometime  in  the  Inline 

(2)  Input  Devices 

II  there  is  any  area  related  to  interactive  graphics  that  has  received  as  much  attention  as  visual  display 
technology,  it  is  input  device  technology.  Input  devices  other  than  the  familiar  keyboard  can  he  classified 
according  to  whether  they  lend  themselves  chiefly  to  pointing,  positioning,  or  drawing  (Newman  A Sproull 
1973).  Most  of  them  are  better  at  one  of  these  than  the  others.  Therefore,  while  a collective  capahilitv  lor 
all  three  basic  human  actions  exists  (pointing,  positioning,  and  drawing),  use  of  any  one  input  device 
usually  results  in  a degraded  capability  for  one  or  two  of  these  actions. 

(a)  Light  Pen 

Possibly  the  most  commonly  used  input  device  is  the  light  pen.  This  is  principally  a pointing  device 
consisting  of  a photocell  and  optica)  system  which  focuses  into  it  any  light  in  the  pen’s  field  of  view,  A 
switch  on  the  pen  is  used  to  shutter  light  to  the  photocell  when  desired. 

Output  from  the  pen  is  amplified  and  fed  to  a flip-flop  which  can  be  read  and  reset  by  the  computer 
Thus,  when  the  pen  is  used  for  pointing,  the  flip-flop  must  either  be  tested  during  the  refresh  cycle  alter 
displaying  each  point  or  sensed  through  an  interrupt  scheme.  The  tirne  required  for  the  pen  to  set  the 
flip-flop  becomes  critical,  because  if  this  time  exceeds  the  time  it  lakes  to  execute  a display  instruction 
(typical  situation),  incorrect  data  will  result  regarding  the  display  element  at  which  the  pen  is  pointed 
(Newman  & Sproull,  1973).  Therefore,  most  of  the  work  in  light  pen  technology  concerns  attempts  to 
achieve  faster  response  times  from  the  pen.  One  technique  is  to  use  photomultiplier  tubes  (highly  sensitive 
photocells);  however,  they  are  too  bulky  to  be  hand  held  and  the  light  must  be  focused  into  them  using  a 
fiber  optic  pipe,  adding  to  the  complexity  and  cost  of  the  pen.  Another  variation  is  to  use  transistor-type 
devices.  These  arc  small  and  inexpensive  but  have  a slow  response  time  (I  to  2 microseconds).  Therefore 
they  are  suitable  only  for  slower  displays. 

When  used  for  positioning  or  drawing,  the  light  pen  requires  a pen  tracking  program  which 
continually  senses  the  pen’s  position  over  the  screen.  Rapid  movement  of  the  pen  can  cause  the  tracking 
program  to  “get  lost,"  so  special  techniques  are  needed  to  handle  this,  adding  to  computer  loading  and 
software  complexity  ( Rundlc,  1 97 1 ). 

The  low  cost  of  the  light  pen  (SI, 500.  (Machovcr,  1972))  probably  explains  its  popularity  and 
widespread  use  despite  a number  of  disadvantages  associated  with  it.  These  include  (a)  unnatural  feeling 
when  held  in  hand,  (b)  lack  of  precision  due  to  pen’s  aperture  and  varying  distance  from  the  screen,  (c) 
experience  of  fatigue  when  used  with  vertical  CRTs,  (d)  slow  response  time,  (e)  necessity  lor  frequent 
button-pusliing,  and  (0  inconvenience  incurred  by  the  necessity  to  attach  it  by  a cable  to  computer 
interface  equipment  (Dedrecn,  1970). 

(b)  Tablet  Based  Devices 

I Acoustic  Pens  and  Tablets.  Input  devices  which  arc  based  on  acoustic  rather  than  optical 
principles  have  also  been  developed.  Apparently,  the  initial  motivation  for  such  devices  was  the  fact  that 
light  pens  cannot  be  used  with  direct  view  storage  tubes,  and  it  was  desirable  in  general  to  make  the  pen 
position  sensing  independent  of  display  refresh  (Machover.  1972). 

The  earliest  acoustic  device  was  the  Lincoln  Wand  developed  in  196b  (Newman  & Sproull,  1973). 
This  consisted  of  a hand-held  ultrasonic  transmitter  emitting  sound  pulses  picked  up  by  four  microphones 
at  the  four  corners  of  the  screen.  This  allowed  input  in  3 dimensions  by  calculating  the  pen’s  x.y,  and  / 


coordinates  as  a function  of  the  delay  between  sound  pulse  transmission  and  reception  However,  the 
calculations  required  were  complex  and  consumed  too  much  computer  time  to  provide  an  efficient  system 

An  improved  device  is  the  Chaff  Pen  Sonic  Digitizer  available  commercially  through  Science 
Accessories  Division  (Newman  & Sproull,  1 ‘>73 ).  This  consists  ol  strip  microphones  along  two  adjacent 
edges  ol  a tablet.  The  stylus  has  a piece  of  ceramic  mounted  close  to  the  tip.  and  at  regular  intervals  emits  a 
spaik  across  the  ceramic  surface  between  two  electrodes.  Tins  pulse  ol  sound  is  picked  up  by  the 
microphones,  and  two  counters  record  the  delay  between  generation  ot  the  spark  and  reception  ol  the 
sound  pulse.  The  device  also  works  in  3D  and  is  the  most  successful  of  such  devices  now  available  and  the 
only  3D  input  device  commercially  available  (Burton  & Sutherland,  1974)  Its  weaknesses  arc  that  sampling 
is  limited  to  100  per  second,  response  time  is  limited  to  the  (relatively  slow)  speed  of  sound  in  air.  and  its 
accuracy  is  limited  to  I in  500  due  to  air  movement  even  in  a quiet  room  (Burton  & Sutherland.  1 * >74 *. 

2.  l hit  age  (Iradicnt  Stylus/Tablets.  These  devices  are  ones  in  which  the  stylus  is  used  to  sense  a 
jxitential  on  the  surface  of  a tablet  consisting  of  semiconductor  material.  The  x-y  coordinates  of  the  sty  lus 
are  then  determined  by  measuring  this  potential  during  the  time  periods  in  which  it  has  been  applied  to 
successive  horizontal  and  vertical  lines  across  and  down  the  semiconductor  sheet  (Newman  & Sproull. 
1 ‘>73 ) The  jnajor  problem  with  such  devices  is  in  finding  a suitable  tablet  material  with  uniform  sensitivity 
so  that  a reasonable  potential  can  be  developed. 

The  Sylvama  I ablet  ( Teixeira  & Sullen.  I96X)  works  basically  on  the  voltage  gradient  principle  but 
detects  sty  lus  position  in  a slightly  different  manner  The  tablet  consists  ol  a resistive  sheet  between  two 
glass  plates,  and  sheet  signals  are  high  frequency  ac  wherein  the  phase  varies  for  each  stylus  position.  Two 
different  frequencies  arc  used  for  horizontal  and  vertical  signals,  and  the  received  signal  is  filtered  into  two 
frequency  components  whose  phases  then  correspond  to  stylus  coordinates. 

3.  Rami  Tablet.  Ol  all  the  devices  developed  either  experimentally  or  commercially  over  the  last 
decade  or  so.  few  distinctive  improvements  (particularly  regarding  accuracy  and  linearity)  have  been  made 
over  the  Hand  lablet  developed  in  l‘t(>4  (Davts  & i His.  1964).  This  device  was  developed  by  Hand. 
Corporation  and  further  refined  and  marketed  by  Bolt.  Beranek  and  Newman  (Newman  & Sproull.  I ‘>73 ). 
It  consists  of  a 10  by  10  inch  drawing  area  composed  ot  two  planes  ( \ and  y ) ol  1 .024  lines  separated  by  a 
thin  my  lar  sheet,  l aeli  of  the  1 .024  copper  lines  in  both  planes  carries  a unique  digitally  coded  signal  dial  is 
detected  by  the  stylus.  Gray  code  is  used  so  that  the  sequence  ot  pulses  in  adjacent  wires  differs  by  only 
one  pulse  position.  Pulses  received  through  the  stylus  are  amplified  and  sent  to  a decoding  network  where 
(■ray  to  binary  integer  conversion  is  performed.  The  disadvantages  ol  the  Hand  Tablet  are  its  high  cost 
(SI0-  to  SI  5-thousand,  (Machover.  I‘>72)).  high  complexity  . and  the  fact  that  the  stylus  itself  is  susceptible 
to  weat  and  tear  (Newman  & Sproull,  1973). 

(c>  Stylus  Independent  Devices 

In  all  the  devices  discussed  so  far.  a stylus  or  pen  is  required  tor  use  in  conjunction  with  the  display 
surface  or  a tablet  The  desirability  ol  freeing  the  man  from  the  need  to  hand-hold  such  a device  has 
spawned  etlorts  to  develop  techniques  for  detecting  finger  position,  touch,  or  pressure. 

1 Trcssurc  and  Tmicli  Ncusirm  Denies  Iliads  (1969)  provides  a review  of  techniques  developed 
for  touch  or  pressure  sensitive  devices.  Ilic  earliest  approach  w as  to  construct  a grid  of  wires  terminating  at 
tiie  front  surface  ot  the  display  lube  and  delect  a point  ol  touch  by  the  influence  ol  the  body  ’s  capacitance 
on  the  underlying  circuit  lllady  recommended  an  improved  technique  doing  away  with  the  grid  ol  wires 
wherein  surface  waves  are  propagated  on  a transparent  glass  plate,  and  any  object  contacting  The  surface 
reflects  energy  to  the  source. 

51  ore  recently  I’obgec  and  I’aif.x  ( 1972)  described  a pressure  sensitive  device  based  on  the  use  ol  a 
membrane  suspended  ovci  a solid  surlacc.  I lus  technique  and  others  in  the  class  of  pressure-and 
touch  sensitive  devices  stiller  the  disadvantage  ol  incurring  background  noise  from  fingerprints  and 
scratches  on  the  sensitive  surface. 

2 l‘"sm,  >n  .Si  mitin  Ikmi  s Sal  Hunger  ( 1972)  describes  a technique  ("light  gate  field")  wherein  me 
display  screen  is  divided  into  aieas.  each  ol  which  can  be  singled  out  bv  plating  a finger  or  other  object 
c I o sc  to  or  upon  it  Ihe  position  ot  the  finger  is  detected  by  its  interruption  ol  infrared  signals  passing  from 
one  sidi  'l  rlie  screen  to  i semiconductor  light  detector  on  tire  other  In  the  described  application,  the 
screen  was  p.nl ilioned  into  it)  areas  Vs  might  be  obvious,  resolution  is  a kev  limitation  of  the  utility  ot  this 
tec  liruquc 


(d)  Other  Devices 


Other  input  devices  arc  primarily  based  on  the  positioning  of  a cursor  on  the  screen  by  various 
techniques.  The  joystick  and  track  ball  (Davis.  1969)  are  two  such  devices,  and  they  rely  on  visual  feedback 
from  the  displayed  cursor  position  to  compensate  for  their  lack  of  linearity.  The  Stanford  Research 
Institute  “mouse"  (I  ngelbart  & English,  1968)  is  similar  to  the  track  ball  in  principle,  but  uses  wheels 
connected  to  potentiometers  to  provide  a simpler  and  lower  cost  device. 

(3)  Motion  Aids 

In  the  earlier  discussion  of  system  requirements  for  man-computer  symbiosis,  the  potential  need  for 
memory  aids  arose  out  of  the  more  general  requirement  for  status  transceivers.  On  the  computer  side, 
memory  is  generally  recognized  as  small  (in  comparison  with  man’s)  but  precise;  on  the  human  side,  man 
has  a relatively  large  but  imprecise  memory.  Therefore,  improvements  relative  to  symbiotic  system 
development  would  expectedly  be  to  increase  the  computer’s  memory  capacity  and  increase  the  man’s 
precision  and  recall  capability. 

(a)  Computer  Memory  Capacity 

Major  efforts  in  this  area  seem  to  be  concentrated  on  developing  effective  graphics  data  structures  to 
permit  mass  storage  to  be  used  effectively.  I ssentially.  then,  the  emphasis  and  the  problem  are  not  so  much 
to  provide  greater  memory  capacity  as  it  is  to  make  efficient  use  of  what  we  have,  considering  the  serial 
nature  of  the  computer  as  a processor. 

The  scope  of  this  paper  did  not  permit  or  warrant  a survey  ol  graphics  data  structures  One  is 
provided  fairly  recently  in  Williams  (1971).  with  a more  general  review  in  Newman  and  Sprout)  (1973) 
Much  of  the  documented  data  structure  research  appears  aimed  at  more  and  more  effective  means  of 
building  one  type  of  display  file.  This  is  the  structured  file,  wherein  various  picture  entities  are  grouped  in 
alternative  ways  so  that  they  can  be  combined  with  other  entities  In  the  display  processor  m forming  a 
complete  picture.  However,  as  Newman  (1971)  points  out.  there  are  two  other  tvpes  ot  tiles  existing  in 
graphics  systems.  One  is  the  display  file  itself,  which  consists  of  lists  ot  instructions  lot  the  display  u it h all 
graphic  data  scaled  and  transformed  as  required.  The  other  is  a tile  which  allows  the  user  to  build  and 
modify  the  structured  display  file,  figure  (>  illustrates  these  three  files  and  their  respective  positions  in  a 
general  graphics  system. 

The  pon  t of  Newman's  article  is  that  three  separate  types  ol  display  files  are  generally  implemented, 
adding  to  the  memory  and  processor  load  that  we  are  try  ing  to  reduce  and  make  more  efficient,  lie  goes  on 
to  point  out  that  some  efforts  have  been  made  to  reduce  the  display  file  triplication  by  combining  the 
modifiable  display  file  with  the  structured  display  file,  but  that  most  are  imitations  or  variations  ol  the 
same  technique  used  in  1963  in  developing  Sketchpad  (Sutherland.  1963).  Ills  own  recommendation  is  to 
eliminate  the  structured  data  tile  altogether,  lie  claims  (p.  651)  for  one  thing  that  its  existence  discourages 
the  development  of  machine  independent  graphic  languages  because  it  is  specifically  structured  for  the 
display  being  serviced  to  minimize  refresh  time.  As  a result,  it  is  generally  difficult,  if  not  impossible,  to 
convert  graphical  programs  from  one  machine  to  another  A distinct  message  acquiied  from  Newman's 
article  is  that  perhaps  our  efforts  should  be  channeled  toward  more  effective  means  ot  eliminating  display 
files  rather  than  toward  building  new  data  structures  to  handle  the  tiles  as  conventionally  organized 

(b)  Human  Memory  Precision  and  Recall  Capability 

Something  accomplished  in  nearly  every  interactive  graphics  system  is  to  permit  man  to  access  the 
computer's  memory.  This,  in  a sense,  enhances  Ins  own  memory  precision  ami  res  .ill  capability  In  allowing 
him  to  selectively  sample  data  stored  in  the  computer  memory  Hut  what  about  the  problem  of  recalling 
(a)  the  nature  of  the  information  that  should  be  recalled,  oi  lb)  mloim.ition  which  is  specific,  on  a 
short-term  basis,  to  the  continually  changing  status  ot  the  system  iclativc  to  us  goals  (which  may  be 
multiple)''  There  arc  some  types  of  information  which  man  requires  perhaps  only  tor  a minute  or  two  in  a 
given  unique  application  of  the  system  and  which,  therefore,  cannot  be  committed  to  man’s  long-term 
memory  nor  specifically  anticipated  and  preprogrammed  for  storage  in  and  retrieval  Horn  the  computer’s 
memory.  Titus,  except  in  highly  rehearsed  system  applications,  one  would  expect  man’s  short-term  memory 
(STM)  to  play  a principal  role. 

The  problem  arises  from  the  fact  that  man’s  STM  is  extremely  capacity  limited  and,  in  the  absence  ol 
sustained  attention,  loses  its  store  rapidly.  Considerable  research  has  been  perloimed  on  the  chaiacteristics 
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ol  S I M 1 1 ills  ii  I’oxnci.  I '»(>'» ) It  has  been  found  tluit  S I M capacity  is  roughly  7 plus  or  minus  2 items,  and 
tli at  retention  tune  ranges  Iroin  500  milliseconds  to  15  minutes.  dc|>cndiiig  largely  on  the  nature  ol  the 
items. 

I.  Simulated  SIM  Jacks  (l‘)C8)  describes  a unique  method  ol  using  the  computer  to  provide  a 
simulated  STM  lor  the  man  lor  short-term  store  ol  relevant  data.  Ills  duel  leason  lor  a simulated  SIM 
requirement  is  due  to  man’s  processor  characteristics  when  acting’  m the  role  ol  a problem  solver 
(automotive  design  problems  at  Ceneral  Motors  in  this  case)  These  characteristics  include  the  application 
ot  trial  solutions  and  subsequent  development  ot  strategies,  lacks  points  out  that  even  though  man  may 
develop  a strategy,  lie  cannot  usually  apply  it  a second  tune  at  a higher  rate  as  a sequence  ol  operations  In 
tact,  he  may  have  forgotten  how  to  apply  it  at  all  and  need  to  redevelop  it  llms.  lacks'  simulated  SI  M is 
primarily  a short-term  procedure  memory.  I sing  it.  much  ol  the  design  process  becomes  one  ol  selective 
reapplication  ot  recorded  strategies. 

Miller  (|Rb5)  had  earlier  suggested  the  development  ot  SIM  aids  to  compensate  tor  man's  poor 
ability  to  perform  logical  operations  which,  he  said,  was  due  in  turn  to  Ins  limited  STM  capacity  IX'spne 
this  early  suggestion  ( l'H>5)  and  the  very  good  evidence  accrued  by  psychological  research  on  the  need  lor 
such  aids,  the  literature  reveals  little  it  any  active  research  on  simulated  SIM  for  interactive  graphics 
sy  stems  (with  the  sole  exception  ot  Jacks’  l‘>t>8  article). 

(4)  Review  and  Conclusions 

In  way  of  review,  the  symbiotic  system  requirement  leading  to  the  above  survey  of  capabilities  in 
status  transceivers  and  memories  was  that  man  and  computer  need  to  he  able  to  determine  whether  or  not 
information  is  relevant.  The  implied  requirement  was  a need  to  know  the  status  of  the  system  relative  to  its 
goals.  Accompanying  these  needs  was  a requirement  for  an  attention-getting  capability  . 

(a)  ( oniputer-to-Man  Communication 

Ol  all  display  types  it  appears  that  raster  scan  CRT  systems  hold  the  greatest  promise  for  widespread 
use  m symbiotic  systems.  This  is  principally  due  to  the  lower  refresh  rate  requirement  when  interlaced 
scanning  is  employed  and  the  significantly  lower  cost  of  raster  scan  displays,  l ittle  mention  is  made  in  the 
literature  about  attention-getting  techniques  beyond  the  traditional  blinking  of  a display  message.  The 
success  ol  this  method  is  dependent  on  the  assumption  that  the  man  is  looking  at  the  display  at  the  time  his 
attention  is  required. 

(b)  Man-To-Computer  Communication 

Most  input  devices  are  designed  primarily  for  pointing  or  positioning  and  drawing,  but  none  are 
optimal  simultaneously  for  all  three.  Of  all  devices  available,  the  Rand  Tablet  developed  in  !lh>4  is 
apparently  the  most  accurate  and  linear;  however,  it  is  also  one  of  the  most  expensive  and  the  sty  lus  is 
susceptible  to  wear  and  tear.  Stylus-independent  devices  have  primarily  been  geared  to  providing  a pointing 
capability,  but  resolution  is  a chief  limitation.  One  of  the  least  expensive  devices  is  the  light  pen.  hut  its 
response  times  are  relatively  slow. 

( c > Memory  Aids 

The  only  significant  work  in  memory  aids  for  the  man  appears  to  be  that  done  by  Jacks  in  l‘>b8  in 
the  development  ol  a simulated  STM.  However  his  work  was  application  oriented  and  included  only 
procedure  memory  for  use  in  automotive  design  work.  Work  on  developing  memory  aids  for  the  computer 
lies  primarily  in  the  development  of  improved  data  structures.  However  it  has  been  suggested  that  efforts  be 
channeled  instead  to  reduction  of  die  number  of  display  files  required  in  the  system 

(d)  Conclusions 

Display  technology  is  generally  adequate,  but  few  if  any  applications  have  incorporated  reliable 
means  of  attention  getting.  Raster  scan  CRTs  are  preferable  due  to  the  lower  refresh  rates  required  using 
interlaced  scan  however,  these  CRTs  necessitate  scan  conversion  and,  so.  are  not  without  dieir  own 
disadvantages  I Vs  pile  an  enormous  amount  of  research,  little  definitive  progress  has  been  made  since  1%4 
in  providing  improved,  flexible,  low-cost  input  devices.  Devices  which  appear  best  suited  tor  man’s  use  m 
transmitting  status  information  are  poorly  adapted  to  use  as  attention-getting  devices.  little  relevant  work 
has  been  done  to  improve  man’s  limited  STM,  despite  considerable  evidence  of  its  need  that  has  been 
accrued  through  psychological  research.  Techniques  for  improving  the  computer’s  memory  capacity  are 
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largely  geared  to  enhancing  data  retrieval  speed  through  the  development  of  better  graphic  data  structures. 
Potential  benefits  may  exist  in  efforts  to  reduce  the  number  of  display  files. 

(Iraphies  Techniques.  Graphics  techniques  may  be  classified  according  to  whether  they  are  geared  to 
twiv  or  three-dimensional  data  representation.  These  classes  are  not  mutually  exclusive  because  most  21) 
techniques  are  useful  as  well  in  31)  picture  portrayal.  However  3D  techniques  arc  in  a class  by  themselves 
due  to  the  special  problems  encountered  in  achieving  perspective  and  depth  cues. 

( 1 ) Tw ’-Dimensional  Teehniq  ues 

Two-dimensional  techniques  include  basic  operations  on  data  such  as  rotation  and  translation, 
pen-tracking  techniques  to  permit  drawing  on  the  screen  with  a light  pen,  scan  conversion  methods  for  use 
with  raster  scan  displays,  and  visual  resolution  processing,  which  is  a data  reduction  technique  recently 
applied  in  interactive  graphics. 

(a)  Basic  Operations 

Translation,  rotation,  scaling,  and  zooming  involve  the  application  of  simple  transformations  which 
can  be  concatenated  to  achieve  desired  goals.  Scissoring,  commonly  done  with  hardware,  involves  selective 
intensification*  of  points  only  when  they  lie  within  the  screen  boundary.  Newman  and  Sprout)  ( l‘)73)  point 
out,  however,  that  scissoring  still  consumes  time  to  trace  invisible  portions  and  can  increase  the  probability 
of  dicker.  The  alternative  is  to  clip  the  data  (restrict  it  to  the  screen  boundaries)  before  display,  and  this 
can  be  done  either  with  hardware  or  software.  All  of  these  basic  operations  are  straightforward  and  well 
developed,  posing  no  noteworthy  technical  problems 

(b)  Pen  Tracking 

One  of  the  tasks  which  cannot  be  performed  using  hardware  is  tracking  the  light  pen  when  it  is  used 
to  generate  drawings  ( Bundle,  l‘)7l ).  This  is  not  as  simple  a problem  as  it  might  seem,  because  the  light  pen 
is  dependent  on  light  from  the  CRT  to  permit  it  to  deliver  position  data.  Therefore,  the  display  system 
must  provide  a source  of  light  for  the  pen  which  follows  the  pen  when  it  is  moved  across  the  screen.  If 
moved  too  rapidly,  the  tracking  program  may  lose  the  pen  completely,  and  a search  for  it  is  necessitated. 

The  typical  approach  (incorporated  in  pen-tracking  software  provided  by  the  manufacturer)  is  to 
generate  a source  of  light  in  the  form  of  sequentially  drawn  radial  lines  from  a center  position  (Davis. 

Any  movement  of  the  pen  away  from  the  center  causes  it  to  intercept  one  of  the  radial  lines,  cueing 
tlie  tracking  program  to  the  propet  direction  in  which  to  move  the  light  source.  If  the  pen  gets  lost,  the 
usual  relocation  method  is  to  begin  a spiral  search  from  the  last  known  position. 

(c)  Scan  Conversion 

To  use  a raster  scan  CRT  system,  some  means  is  required  to  convert  the  x-y  representation  of  data 
into  scan  line  format.  This  operation  is  known  as  scan  conversion,  and  it  e.  n be  performed  using  analog 
hardware  or  software  (Noll,  1071 ).  Scan  conversion  hardware  is  expensive  and  complex.  One  technique,  for 
example,  is  to  use  a conventional  CRT  driven  by  x-y  coordinates  from  the  computer  as  an  intermediate 
output.  The  face  of  this  CRT  is  then  scanned  electronically  to  accomplish  the  required  conversion.  Due  to 
the  reliability  problems  and  expense  of  such  equipment,  scan  conversion  is  usually  performed  by  software 
(Noll,  l‘)7l ). 

The  conventional  method  of  software  scan  conversion  is  to  produce  a direct  binary  image  of  the 
entire  display  from  the  x-y  coordinate  description  of  the  picture  (Jordan  <V  Barrett,  ll>73;  Noll,  IV7I).  This 
image  can  then  be  stored  in  core  lor  line  by  line  read-out  to  the  display  or  on  some  peripheral  storage 
device  interfaced  to  the  display  (e.g.,  drum).  The  latter  method  was  used  in  the  first  application  of  raster 
scan  displays  in  interactive  graphics  (Opliir  et  al.,  l')68).  The  disadvantage  of  this  technique  is  the  sacrifice 
ol  efficiency  resulting  Irom  the  access  latency  of  the  drum  used  for  storing  the  binary  image.  The  former 
method  (core  storage)  was  used  in  what  appears  to  be  the  second  application  of  raster  scan  displays  in 
interactive  graphics  (Noll.  1 ‘>7 1 ).  The  disadvantage  of  this  approach  is  that  random  access  memory  is 
relatively  expensive  and  storage  requirements  are  large.  I or  a screen  size  of  512  by  5 1 2 dots,  2(>2,14-t  bits 
are  needed  to  store  an  entire  binary  image.  Also,  as  Noll  ( l')71 ) points  out.  this  figure  would  be  tripled  if. 
say,  two  additional  bits  were  used  to  determine  the  intensity  of  each  point  to  provide  four  brightness  levels. 
Thus,  a major  problem  in  the  conventional  scan  conversion  method  is  the  amount  of  storage  required  for 
the  binary  image. 
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•1  sco Mi/  problem  is  the  time  required  to  perform  the  conversion  Although  the  calculation  itself  is 
straightforward,  it  must  he  performed  for  each  x-y  coordinate  pair  In  Noll's  (|97l)  application,  the 
calculation  for  each  pair  required  200  microseconds  on  a DDP-224  computer.  To  scan  convert  a picture 
tilling  the  equivalent  of  one-third  of  a 512  by  512  dot  screen,  about  1716  seconds  would  be  required 
fortunately,  as  Noll  points  out.  it  is  not  always  necessary  to  update  the  entire  picture  and  do  a complete, 
new  scan  conversion.  Often  what  is  needed  is  just  a modification  of  the  scene  already  being  displayed  Ilus 
leads  to  a third  problem  associated  with  the  conventional  scan  conversion  technique 

Hie  problem  arises  from  the  necessity  to  delete  some  portion  of  the  currently  displayed  picture  Noll 
( 1**7 1 » provides  a classic,  often  referenced  example  of  the  problem  that  arises  by  presenting  a hypothetical 
display  situation  in  which  a single  line  and  a dot  are  displayed.  Tire  dot  is  moveable  hy  the  user  via  two 
knobs  I ach  time  the  dot  changes  position,  its  previous  position  on  the  screen  must  be  blanked  and  the  new 
one  illuminated  It  the  background  were  initially  made  black  by  placing  all  zeros  in  the  binary  image,  then  a 
0 must  Ire  placed  in  the  previous  dot  position  to  erase  it.  and  a I would  have  to  be  placed  in  the  new 
position  to  illuminate  it  The  problem  arises  when  the  dot  crosses  the  line,  in  which  case  tins  technique 
would  result  in  removal  of  one  dot  from  the  line  itself,  for  this  reason,  special  precautions  are  required 
when  deleting  existing  picture  elements  using  a conventional  scan  conversion  process  wherein  a direct 
binary  image  is  maintained.  Thus,  difficulty  in  picture  modification  may  be  considered  a third  disadvantayi 
of  this  technique. 

A method  has  been  proposed  by  Jordan  and  Barrett  (1973)  to  reduce  the  memory  requirement 
associated  with  the  conventional  scan  conversion  method.  Their  scan  conversion  technique  to  accomplish 
this  is  a modification  of  one  commonly  used  in  conjunction  with  hardcopy  systems  where  high  read-out) 
rates  are  not  required.  In  these  systems,  a small  buffer  area  is  used  over  and  ovet  to  produce  sequential 
groups  of  scan  line  images  (e.g..  10  scan  lines  at  a time).  I ach  time,  a complete  image  is  generated  trout  the 
x-y  display  file,  but  only  those  points  corresponding  to  the  part  of  the  image  to  be  plotted  are  stored  in  the 
buffer.  The  method  developed  by  Jordan  and  Barrett  follows  a similar  procedure  but  is  taster  because  it 
only  requires  the  binary  image  to  be  generated  once  rather  than  every  time  a new  group  of  scan  lines  is  to 
be  processed. 

The  Jordan  and  Barrett  procedure  begins  by  sorting  the  \-y  display  file  according  to  the  scan  line 
group  in  which  each  line  will  first  appear  in  a top-to-bottom  scan.  Hie  results  ot  this  sort  are  stored  in  a 
linked  list  data  structure  in  which  each  list  item  (a  single  line)  is  represented  by  the  beginning  and  ending 
coordinates  of  the  line  and  an  additional  x coordinate  used  lor  maintaining  the  current  \ value  as  the  line  is 
traced  through  a given  set  of  scan  lines.  In  processing  the  linked  list,  a dot  generating  algorithm  is  applied  to 
each  line  in  the  current  scan  line  group,  terminating  il  the  line  leaves  that  group  by  updating  the  current  \ 
coordinate  value  and  attaching  the  list  item  to  the  next  scan  line  group  by  changing  the  link  II  the  line  is 
terminated  within  the  present  scan  line  group,  the  corresponding  list  item  lor  that  line  is  removed  trom  the 
active  list,  and  processing  is  begun  on  the  next  item. 

Speed  and  storage  requirements  were  analyzed  by  Jordan  and  Barrett  1 1973)  using  a I’DI’-S  computet 
Using  16  scan  lines  per  scan  line  group,  processing  time  was  only  slightly  greater  than  that  required  using  a 
conventional  scan  conversion  technique  on  the  entire  raster  (e.g..  about  J.1'  seconds  longer  lot  a total  line 
length  of  100  raster  units,  and  about  1.25  seconds  longei  lor  a line  length  ol  40  raster  units)  However, 
memory  requirements  were  only  about  20 rl  of  that  needed  using  the  conventional  technique  I lie  authors 
conclude  that  a choice  of  16  or  32  for  the  scan  line  group  size  w ill  result  in  at  least  an  SO  reduction  in 
storage  requirements,  with  less  than  100  increase  in  execution  time. 

The  problem  of  picture  modification  was  also  attacked  by  Jordan  and  Barrett  in  another  etlort 
(1974).  They  developed  a design  for  a cell  organized  raster  scan  display  in  which  lines  (vectors)  as  well  as 
characters  can  be  processed.  Conventional  low-cost  alphanumeric  displays  aie  based  on  a cellular 
partitioning  of  the  screen,  each  cell  being  large  enough  to  display  a single  character  ilus  allows  the 
character  generator  to  be  implemented  with  a read-only  memory  which  stores  the  dot  patterns  lot  all 
alphan utnc tics.  On  a cell  by  cell  basis,  the  generator  fetches  the  display  command  Horn  relresh  memory 
and,  based  on  the  command  itself  and  the  current  scan  line  being  processed,  retrieves  one  tow  ol  the 
characters  from  memory.  The  generator  continues  to  process  one  row  ol  each  cell  in  the  scan  line  before 
returning  to  the  next  row  ol  the  first  cell.  Hi  ns,  as  Jordan  and  Barrett  note,  each  cell  is  processed  sevetal 
times  before  it  is  completely  displayed. 
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Ilicn  technique  includes  expanding  the  read-only  memory  to  include  line  segments  as  well  as 
alphanumeric  ch.iiaclcrs.  and  placing  the  refresh  memory  in  Iron!  of  the  character  and  vector  generator  to 
per  nut  modiliialion  ol  pielme  elements  in  then  original  x-y  configuration.  (This  differs  from  the 
implementations  discussed  pieviously  (Noll.  1 ‘>7 1 . Opliir  et  al.  1968),  where  the  character/vector  generator 
pieeeded  the  iclrcsh  memory  .1  lo  implement  the  line  segment  capability  in  memory  , a minimal  set  ot  basic 
patterns  was  derived  which,  when  processed  by  the  vector  generator,  can  produce  a line  segment  of  any 
length  and  orientation,  lo  make  thus  possible,  the  generator  itself  included  capabilities  for  reflection  about 
the  \ axis,  translation,  shilling,  masking,  and  union  (binary  OKing)  ol  patterns. 

I wo  alternative  scan  vonvcision  techniques  lor  this  proposed  display  system  were  developed  (Barrett 
A Ionian.  I'>74)  One  technique  is  ail  extension  ol  the  method  described  previously  (Jordan  A Barrett. 

1 1>7 si.  which  implements  processing  ol  all  lines  in  one  cell  before  proceeding  to  the  next.  Thus,  in  addition 
to  transferring  the  current  x coordinate  value  lor  a line  to  the  next  scan  line  group  when  required,  the 
technique  operates  on  a cell  by  cell  basis  and  transfers  the  current  x coordinate  value  to  the  next  cell  as 
required.  I his  is  termed  a presorting  algorithm  since,  as  described  earlier,  the  lines  of  a drawing  are 
presoited  according  to  the  row  in  which  each  line  will  first  appear  in  a top-to-bottom  scan  ( i.e..  sort  lines 
'..'dictating  display  commands).  The  authors  point  out  (Barrett  A Jordan.  1 ‘>74 ) that  this  method  is 
not  well  suited  to  interactive  graphics,  where  a small  number  ol  screen  elements  are  modified  frequently. 
The  reason  is  that  the  entire  x-y  representation  of  the  picture  must  be  available  before  the  scan  conversion 
pioccss  begins,  due  to  the  technique  of  tracing  each  line  to  its  termination  point  to  record  all  endpoints 
prior  to  starting  the  scan  conversion.  Therefore,  to  modify  a picture  the  x-y  representation  must  first  be 
updated,  followed  by  a complete  scan  conversion. 

I he  alternative  method  is  termed  a postsorting  algorithm  wherein  display  commands  for  an  entire  line 
(vector)  id  the  picture  are  generated  first,  then  placed  into  their  proper  positions  in  a cell  display  file  (i.e., 
sort  display  commands  after  they  are  generated  for  a given  line).  This  allows  commands  representing  any 
given  line  to  be  added  or  deleted  without  affecting  the  commands  for  other  elements  of  the  picture.  Thus, 
it  is  well  suited  to  an  interactive  graphics  situation  but.  as  the  authors  note,  requires  a more  complex  data 
structure. 

An  analysis  ol  execution  times  and  memory  requirements  was  performed  for  the  two  cell  oriented 
scan  conversion  techniques  and  compered  with  those  of  the  conventional  technique  (Barrett  A Jordan, 
l‘>74).  A summary  ot  key  results  is  | resented  in  figure  7.  fire  presorting  technique  took  slightly  longer 
than  the  conventional  technique  but  required  less  memory  roughly  the  same  comparative  results  as  those 
obtained  in  the  earlier  study  (Jordan  A Barrett,  1974).  The  time  required  to  add  or  delete  a line,  however, 
was  significantly  greater  than  for  either  the  lonvcntional  or  postsorting  techniques.  The  conclusion  was  that 
the  presorting  algorithm  is  suitable  for  static  display  updating  but.  as  anticipated,  not  well  suited  for 
interactive  graphics  applications. 

The  postsorting  technique  required  considerably  more  time  than  either  of  the  other  two.  for 
example,  on  a C’l X ’ MOD  computer  scan  conversion  ot  a total  line  length  ol  120  units  of  raster  width 
(equivalent  lo  about  2.VI  ol  a 512  by  512  dot  screen ) required  about  24  seconds  using  the  postsorting 
method,  and  between  5 and  5 seconds  lor  the  other  two  methods.  Memory  requirements  for  the  postsorting 
method  exceeded  those  using  presorting.  They  were  less  than  those  using  the  conventional  technique  only 
for  total  line  lengths  not  exceeding  about  125  units  of  raster  width,  at  which  point  and  beyond  they 
equalled  or  exceeded  the  memory  requirements  using  the  conventional  method  The  time  to  modify  a line 
using  postsorting  was  slightly  greater  than  that  using  the  conventional  method  for  line  additions,  but 
significantly  less  for  line  deletions.  This,  of  course,  is  where  the  conventional  method  encounters  one  ol  its 
major  problems  I oi  example,  times  to  add  and  delete  a line  in  a picture  containing  a total  line  length  of  45 
units  ol  raster  width  were  about  1.55  seconds  and  .0.1  seconds,  respectively,  for  the  conventional  method. 
But  the  time  required  using  the  postsorting  method  was  only  about  I <>  seconds  for  both  addition  and 
deletion,  finis,  the  authors  conclude  that  the  postsorting  scheme  represents  a significant  improvement  in 
interactive  capability  over  the  conventional  raster  scan  conversion  technique. 

(d)  Visual  Resolution  Processing 

All  ot  the  21)  techniques  discussed  up  until  now  have  represented  operations  on  graphic  data,  largely 
to  transform  it  into  a desired  representation  lor  display.  Bussell  and  Taxin  (1972)  have  described  an 
additional  technique  which  is  aimed  at  reducing  the  quantity  ol  data  to  he  displayed  without  noticeably 
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Figure  7 Summary  of  key  results  of  Barrett  and  Jordan  experiment  * 


•Copied  from  (Harrctt  A Jordan,  1974,  pp.  161.  162) 


degrading  displa>  quality  Their  technique  is  called  visual  resolution  processing  because  il  is  based  on 
elm  mating  from  a scene  to  be  displayed  those  vectors  which  cannot  be  visually  resolved. 

Use  of  then  technique  begins  with  the  establishment  ol  a minimum  resolvable  element  (\1  Kl  ),  which 
is  the  length  ol  the  diagonal  of  a rectangle  whose  sides  are  equal  to  the  x and  y axis  resolution  factors  ol  the 
display.  Any  vector  whose  length  is  less  than  MK1  and  which  does  not  differ  from  its  predecessor  iri 
duection  by  an  angle  greater  than  alpha  (alpha  usually  experimentally  chosen)  is  then  discarded.  The  angle 
test  is  required  to  preserve  connectivity  of  the  display.  Thus,  the  process  simplifies  the  display  and  reduces 
data  by  stepping  along  a given  connected  line  drawing  and  replacing  series  of  short  vectors  (having  negligible 
directional  differences)  with  single  longer  vectors. 

Tests  were  performed  on  a display  of  6 alphabetic  and  special  characters  (A,B,( ) using  MR! 
values  ranging  from  0 to  .X.  Results,  which  were  subjectively  evaluated  by  Russell  and  laxin.  are  shown  in 
figure  X.  A 78'J  reduction  in  the  number  of  vectors  was  possible  while  still  maintaining  an  excellent, 
distortion-tree  display.  The  preprocessing  required  added  about  20 - to  the  total  vector  preparation  time. 
I h us.  the  technique  can  be  used  to  effect  a tradeoff  between  time  and  the  number  of  vectors  or  lines 
comprising  a scene. 
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/ nrnrt  s Results  of  Russell  and  Taxin  experiment 


A point  worth  noting  is  the  potential  effect  of  visual  resolution  processing  in  a system  employing  the 
scan  conversion  algorithms  of  Barrett  and  Jordan  ( I *>74)  described  earlier.  In  that  study,  it  was  found  that 
the  execution  time  for  the  algorithms  is  a function  of  the  total  length  of  lines  in  the  drawing,  but  is  little 
affected  by  the  number  of  lines  (Rarrett  Si  Jordan,  1 ‘>74 , p.161).  Therefore,  it  may  be  inferred  that  visual 
resolution  processing  would  be  of  little  benefit  in  the  cell  organized  raster  display  system  of  Barrett  and 
Jordan.  It  would  probably  have  its  greatest  utility  in  a conventional  random  scan  (R  I system.  In  either 
case,  however,  it  could  be  used  to  appreciably  reduce  memory  requirements  for  storing  the  x-v  display 
image.  The  real  value  of  visual  resolution  processing  can  only  by  hypothesized  on  the  basis  of  the  relatively 
simple  tests  performed  (Bussell  .V  laxin.  1 ‘>7 2 ).  and  the  above  remarks  should  be  interpreted  accordingly. 


(2)  Three  Dimensional  Techniques 

Psychologists  estimate  that  on  a single  dimension  men  are  capsule  01  discriminating  7 plus  or  minus  2 
objects  (David.  1967).  I'll  is  is  relevant  to  symbiotic  system  design  because  large  quantities  of  data 
representing  system  status  may  need  to  be  transmitted  to  the  man  At  the  same  time,  display  clutter  must 
be  avoided,  and  display  quantity  limited  to  what  the  man  can  expectedly  discriminate  and  assimilate  David 
( 1967)  points  out  that  multidimensional  displays  can  help  considerably  in  presenting  quantities  of  data  in  a 
form  interpretable  and  usable  by  man.  Therefore,  3D  techniques  are  of  particular  in  iciest. 

Newman  and  Sproull  (1973)  outline  three  basic  techniques  for  achieving  71)  effects  (a)  display  ol 
several  orthogonal  views  of  the  object.  ( b ) (X’rspcctivc  displays,  or  wire  frame  drawings,  and  ( c > 3D  displays 
involving  hidden  line  and  surface  removal.  These  are  listed  in  increasing  order  ol  their  general  desirability 
The  first  two  techniques  present  no  real  technical  problem,  and  their  use  is  commonly  encountered  in 
applications  articles  in  the  literature.  Ophir.  Shepherd,  and  Spinrad  (1969)  describe  an  applied 
enhancement  ol  perspective  displays  by  making  them  stereographic  as  well. 

The  third  technique,  for  production  of  3D  displays  with  hidden  line/surface  removal,  is  not  at  all  a 
simple  task  and  has  received  considerable  attention  in  the  past  10  to  12  years.  Research  efforts  have 
explored  various  means  of  achieving  depth  cues,  including  hidden  line  and  surface  removal,  shading  ol 
objects,  and  variation  of  texture.’  I he  fundamental  problem,  however,  appears  to  be  the  efficient  removal 
of  hidden  parts,  because  the  success  of  the  other  techniques  is  largely  dependent  on  this  capability  In  1 966 
Sutherland  identified  the  hidden  line  problem  as  one  of  ten  that  were  then  unsolved  in  computer  graphics 
Seven  y ears  later,  Newman  and  Sproull  ( 1073)  stated  that  contemporary  hidden  line  and  surface  algorithms 
were  not  optimal,  each  having  its  own  advantages,  disadvantages,  and  inefficiencies.  Therefore,  this  problem 
appears  to  be  the  most  immediately  critical  to  the  effective  application  ol  31)  techniques  in  symbiotic 
systems. 

(a)  Hidden  Linc/Surface  Techniques 

The  first  practical  solution  of  the  hidden  line/surface  problem  was  developed  in  I on.)  by  I . (>. 
Roberts  at  MIT  Lincoln  Laboratory.  The  Roberts  algorithm  tests  each  line  in  the  picture  against  every 
opaque  surface  to  discover  what  parts  of  it  will  be  visible  (Newman  A Sproull.  1973;  Sutherland.  Sproull,  A 
Schumacker.  1973).  With  minor  distinctions  in  technique,  several  other  algorithms  based  on  the  same 
principle  sprang  up  in  the  late  sixties  and  early  seventies  (Appel.  1967;  (lalimberti  A Montanari,  1969. 
Loutrel,  1970).  According  to  Sutherland  et  al.  (1973),  all  such  algorithms,  sharing  the  requirement  that 
every  object  be  compared  with  every  other  object,  suffer  from  long  computational  time  requirements  that 
preclude  their  use  in  complex  graphics  applications.  C .cue rally , the  computation  cost  grows  as  the  square  ol 
the  number  of  objects  in  the  environment. 

The  mid-sixties  produced  a surge  of  “brute  force”  techniques  characterized  by  then  test,  on  a line  by 
line  basis,  of  every  point  comprising  every  line  (Appel,  1967).  Their  main  disadvantage  is.  again,  long 
computation  times  even  lor  moderately  complex  objects  ( Appel.  1967;  Loutrel,  1970)  Appel  ( 1967)  notes 
for  instance,  that  on  an  IBM  7094,  the  computation  time  for  a scene  consisting  ol  40  surfaces  and  I 50  lines 
can  exceed  IS  minutes!  This  would  clearly  be  unsatisfactory  in  any  symbiotic  system. 

According  to  Wright  (1972,  1973),  the  algorithm  most  widely  used  was  developed  in  |9oX  In  li. 
Kubert.  His  method  is  in  many  ways  a brute  force  approach  bill  takes  advantage  ol  the  observation  that  any 
line  segment  from  the  point  of  regard  to  a visible  point  on  the  surface  contains  only  the  visible  point 
(Kubert,  Szabo,  & (iiulieri,  1968).  Line  segments  directed  to  invisible  points  contain  more  than  one  point 
on  the  surface.  By  capitalizing  on  this  observation,  Kubert  \ algorithm  tests  each  point  loi  visibility  in  a 
relatively  efficient  way.  However  it  is  only  applicable  to  single  valued  functions  ol  two  variables,  i e..  z 
f(x.y)  (Wright.  1972).  Speedwise,  the  computing  time  on  a ('DC  <><>00  grows  proportionally  to  Vcubed. 
where  the  surface  is  represented  by  an  N by  N height  airay  (Wright.  1 973). 

Sutherland  et  al.  ( 1973)  notes  that  hidden  line/surface  algorithms  may  be  categorized  into  two 
classes,  depending  on  whether  they  involve  a search  of  the  object  space  (one  class)  or  the  image  space  (the 


’Newman  and  Sproull  ( 1973)  believe,  however,  that  the  depth  cueing  effects  used  in  computer  in  opines  have  been 
traditionally  chosen  more  for  their  expediency  than  for  the  accuracy  of  their  effect 


second  class)  (>b|ccl  space  algorithms  examine  each  oh|ect  to  discover  which  parts  will  he  hidden  by  other 
parts  oi  oilier  nh|ects.  * )nl s then  is  an  image  space  created  Iroin  the  visible  object  parts.  Idle  image  space 
.tleorillimx.  howcvci.  concern  themselves  only  with  what  the  image  will  he  at  each  resolvable  dot  on  the 
display  sciccn. 

Idle  techniques  ol  Roberts  and  others  discussed  abuse  ( Appel.  1 ‘>(>7 ; <>alimbcr!i  & Montanan.  1969, 

I oiitrel.  l*»7t»  ate  examples  ol  object  space  algorithms  and  sutler  from  this  disadvantage  of  long 
, nmpiitatinii  tunes,  (icuc  rally.  the  image  space  technn|iies  appear  to  hold  the  greater  promise. 

1 Image  Spun  Ugorithms.  Image  space  teclmii|iies  may  be  classified  according  to  (a)  when,  in  the 
ordei  ol  the  s.iuoir  compulations,  the  visibility  test  is  performed:  and  (b)  the  order  ol  sorting  along  the 
v.\  and  i axes  (Sutlieilatid  et  al..  I * * 7 s V Depth  priority  techniques  postpone  the  visibility  test  until  last  and 
base  it  on  a computation  ol  the  depth  ol  vaiinus  surfaces  as  observed  from  the  point  of  regard.  List  priority 
algorithms  essentially  perform  the  visibility  test  first  by  ordering  all  surfaces  into  the  form  of  a list 
according  to  their  respective  depths  Within  each  type  (depth  and  list  priority),  data  may  be  sorted  in 
various  orders  with  respect  to  the  three  axes,  but  certain  orderings  appear  to  predominate  depending  on  the 
computational  position  (order)  of  the  visibility  test. 

Depth  I’rrorily  Techniques 

Depth  priority  techniques  can  be  further  subdivided  into  those  which  sample  areas  of  the  screen  (area 
sampling  methods)  and  those  which  sample  points  ol  the  screen  (scan  line  methods). 

Area  Sampling  Algorithms.  An  algorithm  developed  by  J.  I . Warnoek  at  University  of  Utah  in  1969  is 
cited  as  a representative  example  of  area  sampling  and  is  described  by  Sutherland  et  al.  (1973).  The 
Wamock  algorithm  is  based  on  a search  lot  homogeneous  w indows  on  the  image  surface,  each  of  which,  in 
their  entirety,  can  be  displayed  oi  considered  hidden.  A window  (rectangle  in  the  xy  plane)  is  considered 
lion  ogeneotis  it  (a)  no  faces  fall  within  it  at  all.  or  (b)  one  face  completely  covers  it.  In  the  first  instance, 
there  would  simply  be  nothing  to  display  in  the  window,  or  perhaps  it  would  be  shaded  to  represent 
background.  In  the  second  instance,  the  window  is  display  ed  tl  the  covering  lace  is  nearer  the  viewpoint 
than  every  other  face  that  falls  m the  window.  In  the  event  a window  is  intersected  by  a face,  it  is 
subdivided  into  four  smaller  windows  and  each  is  retested.  Ibis  subdivision  continues,  as  necessary,  down 
to  the  si/e  of  a raster  element,  terminating  earlier  only  when  a window  is  determined  to  be  homegeneous. 
Sutherland  et  al.  ( 1973)  points  out  that  a major  problem  with  this  method  is  that  it  is  not  easily  used  with 
raster  scan  CRTs,  since  decisions  about  various  windows  are  not  reached  in  a left  to  right,  top  to  bottom 
order. 

Wright  ( 1973)  describes  a technique  in  which  areas  ol  the  screen  representing  lower  boundaries  on  y 
are  successively  created  along  the  / axis.  As  a search  along  the  / axis  proceeds,  any  point  lying  below  the 
greatest  lower  boundary  thus  far  established  is  declared  invisible.  Thus,  beginning  at  the  point  closest  to  the 
observer  and  moving  away,  no  line  is  drawn  which  is  below  a line  that  has  already  been  drawn.  Wright's 
(1973)  method  was  compared  with  Kubert’s.  It  was  shown  that  on  a ( DC  6600.  computing  time  for 
Wright's  technique  grows  as  N-squarcd  (for  an  N by  N height  array  ).  whereas  Hubert's  method  incurs  time 
growths  proportional  to  N -cubed.  Both  of  these  techniques  may  be  classified  as  (xy)/  sorts,  since  the  xy 
plane  is  lust  considered,  followed  by  a depth  search  along  the  / axis. 

Scan  line  Algorithms.  The  general  philosophy  underlying  scan  line  algorithms  is  the  same,  regardless 
ol  whose  particular  algorithm  is  involved  (Teixeira  & Sallen.  1968).  first,  edges  are  sorted  by  y so  that  only 
those  edges  intersecting  the  current  scan  line  are  examined.  Next,  a sort  along  the  x axis  is  performed  to 
determine  sections  of  each  scan  line  within  each  of  which  the  same  face  may  be  visible  finally , a / sott  is 
performed  among  all  segments  falling  within  each  section  ol  a scan  line  to  determine  which  is  visible  Thus, 
the  techniques  are  characterized  as  yx/  sorts. 

lioiikiiiglit  (1970)  describes  a line  scan  algorithm  particularly  well  suited  to  generation  ol  shaded 
images.  Mechani/atiori  ol  the  x sort  is  performed  by  first  associating  a discrete  variable  (tlag)  with  each 
polygon  comprising  the  scene.  As  the  scan  moves  from  left  to  right,  the  Hag  lot  a polygon  is  Hipped  each 
time  the  scan  crosses  one  of  its  edges.  Only  those  poly  gons  whose  Hags  are  flipped  to  the  “in"  position  lie., 
scan  is  within  those  polygons)  are  used  in  distance  computations  As  each  edge  is  crossed  a decision  is  made 
on  which  ol  the  “in"  polygons  is  closest  to  the  observer,  and  information  is  accordingly  generated  about 
the  intensity  ol  the  scan  from  that  point  up  to  the  next  change  point.  ( Intensity  is  computed  as  a (unction 
ol  the  angle  between  the  surface  ol  the  polygon  and  a line  drawn  Irom  the  light  soutce  to  the  surlace.) 
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Bouknight  ran  tests  ol  his  algorithm  compared  with  tliat  ul  Warnoek  ( homogeneous  windups 
method)  on  a ('IK'  1604  computer  Results  are  summarized  in  Table  2.  Boukmghl  adds  that  the 
implementation  ol  lus  algorithm  occupies  TO  thousand  words  ol  memory.  including  data  sioiatv.  and  is 
capable  of  processing  about  1,000  polygons,  each  with  a maximum  ol  lour  vertices. 

Table  2.  Results  of  Comparison  of  Warnoek  and  lioukiiight  Methods 


Computing  Times3 
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last  Priority  Techniques 

The  first  real-time  solution  to  the  hidden  surface  problem  was  developed  at  (iencral  l icet ric  h\  R.  V 
Schumacker  (Rougelot.  I960;  Sutherland  cl  al..  1973).  The  technique  used  a list  priority  approach  to 
develop  a high  quality  image  presentation  system  for  use  in  visual  lliglit  simulation  which  was  implemented 
at  the  NASA  Manned  Spacecraft  Center  in  1968.  Basinger  ( 1973)  and  Bouknight  (1970)  report  that  the 
NASA  system  (625  scan  lines)  was  developed  for  space  rendezvous  and  docking  simulation  studies;  and  that 
the  basic  system  capabilities  included  up  to  240  edges  ( maximum  40  planes  with  o edges  per  plane ) 

A second  system  (525  scan  lines)  was  delivered  to  the  Naval  Air  Training  facility  at  Kingsville.  lexas 
in  1972  ( Basinger.  1973).  This  system  is  capable  of  displaying  scenes  composed  of  5 1 2 edges  In  addition,  a 
2. 000-edge  environment  is  available  in  the  form  of  a data  base  implemented  on  magnetic  disk,  from  which 
any  512-edgc  scene  may  be  composed  during  the  system's  use  in  an  experimental  pilot  training  program. 

Schumacker's  technique  (zyx  sorting  order)  is  based  on  a concept  which  he  introduced  wherein 
certain  collections  of  faces  are  termed  clusters  and  are  treated  as  a group  (Sutherland  el  al  . 1973).  \ 
collection  of  faces  is  a cluster  only  if  it  permits  a preset  priority  order  ol  visibility  to  be  established  for 
clusters  independent  of  the  view  point.  This  permitted  an  examination  ol  the  relations  between  clusters 
rather  than  between  all  of  the  faces  in  the  scene.  This.  then,  could  he  followed  by  processing  of  only  those 
cluster  sections  which  are  visible. 

Schumacker  restricted  the  environment  to  ensure  that  clusters  are  all  linearly  separable.  Cluster 
priorities  were  computed  by  developing  a tree  whose  nodes  represent  planes  separating  the  various  elusleis 
and  whose  two  node  branches  represent  the  side  ol  the  plane  on  which  the  viewpoint  is  located,  figure  9 
gives  a simplified  example  of  the  prioritizing  technique.  I ssentially.  the  Schumacker  method  is  to 
precompute  as  much  priority  ordering  information  as  possible  about  the  environment  before  ever  producing 
any  pictures.  Therefore,  his  technique  is  well  suited  to  the  static  environment  found  m (light  simulation, 
but  not  well  suited  to  environments  in  which  many  objects  move  independently  (Sutherland  et  al  1973) 

2, 000-Id ge  Computer  Image  deaeration  (Cl(!)  System.  Basinger  (1973)  describes  a recently 
developed  system  which  is  capable  ol  displaying  scenes  comprised  ol  up  to  2.000  edges  lot  a wide  angle 
field  of  view  pilot  training  simulation  research  facility.  The  basic  techniques  include  Schumacker's  list 
priority  method  as  well  as  many  enhancements  that,  collectively,  provide  for  increased  processing 
capability.  With  this  system,  it  is  possible  to  simultaneously  display  scenes  foi  two  independently  operated 
cockpits,  each  operating  anywhere  within  a 1 ,250  by  1 .250  nautical  mile  area. 

Tlie  environment  is  modeled  in  the  form  of  a disk-resident  data  base  which  incorporates 
Schumacker's  prioritizing  information  for  the  various  objects  (clusters).  The  data  base  contains  100.000 
edges,  expandable  to  600, 000.  Two  general  purpose  digital  computers  (SI  I Ho.  32  thousand  word  core 
commonly  addressable)  and  one  special  purpose  computer  are  Used  to  process  data  from  the  environment 
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ami  produce  the  required  scenes  for  either  or  botli  of  two  cockpits.  One  SI  I K(>  ( I’l  interlaces  with 
memory  in  the  simulation  computer  to  determine  environment  features  that  are  irt  the  pilot's  field  <>!  view 
and  perform  basic  coordinate  transformations.  The  second  SI  I So  CIM  interlaces  with  the  first  via 
commonly  addressable  memory  and  fetches  required  edge  data  from  the  4 2 M Its  te  dish  containing  the 
environment.  This  edge  data  is  stored  in  core  as  the  active  data  base.  The  special  purpose  computet  ( |i 
cabinets  of  hardware  housing  additional  core  memory  and  digital  processors)  is  interlaced  with  both  general 
purpose  computers  via  their  common  memory,  and  accomplishes  the  actual  generation  ot  video  signals 

Hidden  faces  are  determined  by  scanning  the  environment  with  a vector  extending  from  the 
viewpoint  to  the  various  faces  of  objects,  and  comparing  its  angular  orientation  with  that  ot  a unit  v<  tor 
normal  to  the  surface  of  the  face.  Idges  common  to  adjacent  faces  that  are  both  hidden  are  also  hidden 
This  effects  a rapid  elimination  of  hidden  edges  and  is  a key  feature  in  accomplishing  the  processing  in  teal 
time  (Basinger.  1474).  Another  key  feature  is  included  iti  the  scan  conversion  process  itsell  Inputs  to  the 
converter  include  the  raster  line  at  which  each  edge  first  and  last  appears,  and  the  slope  of  the  edge  in  the 
display  plane.  This  permits  edges  to  be  efficiently  scan  converted  by  building  then  binary  image  recursively 
When  conversions  are  accomplished  for  scan  line  n,  elements  of  edges  which  have  not  vet  terminated  are 
computed  for  scan  line  n+l  using  the  slope  information  ( Beck.  1474) 

The  CIG  system  drives  fourteen  36  inch  CRT  displays,  seven  arranged  around  each  cockpit  to  el  feet  a 
± 120°  horizontal  by  +120°,  -40°  vertical  field  of  view  for  the  pilot.  It  is  part  ol  a simulation  complex 
developed  under  the  direction  of  the  Ait  Force  Human  Resources  laboratory's  Advanced  Systems  Division 
Wright-Patterson  Air  F'orce  Base,  Ohio,  for  use  in  training  simulation  research  It  is  obvious  that  the 
impressive  CIG  processing  capability  is  largely  attributable  to  an  appreciable  amount  ot  computing  pow  r 
devoted  exclusively  to  that  task.  Thus,  while  some  of  the  various  techniques  it  employs  may  merit 
extension  and  further  investigation,  the  cost  of  a large-scale  implementation  such  as  this  would  preclude  its 
widespread  use  in  man-computer  symbiotic  systems. 

2.  Additional  Observations.  The  review  presented  of  hidden  line/surface  techniques  does  not  nearly 
exhaust  the  vast  bibliography  on  the  subject  nor  cover  the  multitude  ol  methods  that  have  been  tried  to 
develop  improvements  here  and  there  on  the  alternative  types  ol  algorithms.  However  it  is  believed  to  have 
fulfilled  its  major  objective  of  outlining  the  basic  approaches  that  are  available  and  some  ol  the  principal 
problems  encountered  with  them. 

Sutherland  et  al.  (1473)  concluded  their  survey  with  the  observation  that  sorting  is.  in  essence,  the 
major  problem  of  hidden  surface  detection.  The  sorting  order,  in  particular,  may  represent  a vet  untapped 
framework  from  within  which  new  methods  may  be  developed  and  explored.  (That  new  methods  are 
required  is  fairly  obvious  in  view  of  the  reported  computation  times,  representatives  of  which  were 
discussed  earlier.)  The  sorting  orders  used  in  the  image  space  algorithms  were 

Area  Sampling  (xy)z 

Scan  Line  yxz 

list  Priority  zyx 

Sutherland  suggests  that  one  avenue  for  new  research  may  be  to  try  other  sorting  ordets 

It  seems,  too,  that  perhaps  an  optimal  sorting  order  does  not  exist  per  se.  but  depends  on  the  nature 
of  the  environment.  The  major  result  of  sorting  is  to  eliminate  information,  and  the  density  ot  information 
along  the  various  axes  could  be  used  to  identify  that  axis  along  which  the  sorting  process  will  eliminate  the 
greatest  amount  of  information.  Therefore,  it  may  be  worthwhile  to  investigate  the  use  ol  variable  sorting 
orders  which  depend  on  the  display  environment  of  the  moment 

( 3 ) Review  and  ( 'onclusions 

The  symbiotic  system  requirement  leading  to  the  above  survey  ol  giaphics  techniques  was  that  both 
man  and  computer  require  techniques  for  formatting  and  presenting  information  (which  may  Iv 
unanticipated  by  the  receiver)  in  a way  that  will  he  clearly  understood  Both  3D  and  graphics 
techniques  were  reviewed  in  light  of  this  requirement. 

(a)  21)  Techniques 

Basic  2D  operations  such  as  rotation,  scaling,  zooming,  etc,  pose  no  distinct  technical  piobleuis.  as 
methods  for  accomplishing  these  operations  are  relatively  straightforward  and  have  been  proven 
Pen-tracking  techniques  are  readily  available;  however,  the  ever  present  possibility  ol  losing  the  |vn'x 
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position  it  it  is  moved  too  rapidly  remains  a problem  tiiat  has  been  handled  so  far  only  by  means  of  a 
systematic  search  of  the  screen. 

Scan  conversion  is  normally  performed  using  software  due  to  the  complexity  of  analog  equipment  for 
implementing  a hardware  capability.  The  conventional  method  of  scan  conversion  (producing  a direct 
binary  image  of  the  display)  has  three  associated  problems  (a)  large  storage  requirement  for  the  binary 
image,  (b)  large  computational  times,  and  (c)  difficulty  in  picture  modification.  Techniques  to  reduce 
storage  requirements  arc  accompanied  by  an  increase  in  computation  time;  however,  the  potential  percent 
reduction  in  storage  is  about  8 times  the  corresponding  percent  increase  in  time.  I.fforts  to  enhance  picture 
modification  capability  are  based  around  cell  organization  of  the  display  and  placement  of  the  refresh 
memory  in  front  of  the  character/vector  generator.  Associated  scan  conversion  techniques  have  been 
developed  for  the  cell  organized  display.  The  technique  seemingly  best  suited  for  interactive  graphics 
applications  achieved  a significant  reduction  in  the  time  required  to  delete  lines  during  picture 
modification.  However,  its  general  processing  time  was  on  the  order  of  4 to  8 times  greater  than  that  of 
previously  developed  techniques. 

A unique  method  of  significantly  reducing  the  -number  of  vectors  comprising  a scene  has  been 
developed  (visual  resolution  processing).  However  the  method  would  have  its  greatest  utility  in  random  scan 
CRT  systems.  In  raster  scan  systems,  it  could  be  used  to  reduce  the  size  of  the  x-y  data  base  but  would  not 
favorably  affect  computational  time. 

(b)  3D  Techniques 

The  most  significant  problem  in  the  use  of  3D  techniques  appears  to  be  the  removal  of  hidden  lines 
and  surfaces.  The  earliest  techniques  developed  as  well  as  the  techniques  most  widely  used  today  employ 
object  space  algorithms  which  examine  each  object  to  discover  which  parts  will  be  hidden  Only  then  is  a 
picture  created  by  deriving  an  image  space  or  display  plane  image  These  techniques  all  suffer  the 
disadvantage  of  long  computation  times. 

Techniques  employing  image  space  algorithms  are  characterized  by  their  attention  to  the  nature  of 
the  display  image  at  each  resolvable  dot  on  the  screen,  rather  than  to  the  total  object  underlying  the  image. 
Depth  priority  image  space  methods  postpone  the  visibility  test  until  last,  and  perform  it  on  the  basis  of 
surface  depth  as  observed  from  the  viewpoint.  These  methods  perform  their  tasks  by  operating  on 
successive  screen  areas  or  on  successive  scan  lines.  The  Waniock  area  sampling  method  is  not  easily  used 
with  raster  scan  CRTs  because  of  the  unsystematic  processing  order  of  screen  areas.  The  Wright  area 
sampling  method  has  a significant  speed  advantage  over  earlier  techniques,  but  at  that,  the  computational 
time  grows  as  N-squarcd,  where  an  N by  N height  array  is  involved.  Scan  line  techniques  are  characterized 
by  a systematic  yxz  order  data  sort  to:  (a)  limit  attention  to  the  current  scan  line,  (b)  determine  sections  of 
the  scan  line  within  which  each  lace  may  be  visible,  and  (c)  determine  which  faces  are  visible  based  on  their 
proximity  to  the  viewpoint  These  techniques  appear  to  be  generally  taster  than  area  sampling  methods,  but 
they  arc  memory  consuming  and  still  require  minutes  ot  computet  time  lor  single,  moderately  complex 
objects. 

last  priority  image  space  techniques  perform  a portion  of  the  visibility  test  first  by  generating  a list  of 
surfaces  ordered  according  to  their  respective  depths.  Schumacker's  method  is  the  best  known  and  has  been 
applied  in  several  large  systems  to  provide  visual  simulation  for  crew  training.  While  achieving  the  real  time 
solutions  required  in  such  applications,  the  method  is  not  well  suited  to  environments  in  which  many 
objects  move  independently.  Large-scale  applications,  such  as  that  described  by  Basinger  (1973),  require 
extensive  computing  power  devoted  exclusively  to  display  generation  to  accomplish  the  job  in  real  time. 

(c)  Conclusions 

Scan  conversion-  methods  are  generally  too  time  consuming  or  too  memory  consuming  to  be 
efficiently  used  in  complex  display  generation  for  man-computer  symbiotic  systems.  I.fforts  to  reduce  time 
or  memory  have  only  resulted  in  a tradeoff  between  the  two.  Hardware  scan  conversion  appears  to  provide 
the  only  readily  available  solution.  However,  the  additional  cost  and  potential  maintenance  problems  make 
this  solution  far  less  than  optimal. 

The  major  problem  in  achieving  31)  effects  appears  to  be  the  elimination  of  hidden  lines  and  surfaces. 
A tremendous  amount  of  work  has  been  performed  in  attempts  to  efficiently  resolve  tins  problem.  The 
image  space  techniques  appear  to  hold  the  greatest  promise  from  the  standpoint  of  minimizing 
computational  time.  However,  they  scent  to  accomplish  this  time  reduction  by  sacrificing  flexibility  (e.g.. 
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they  lack  ability  to  handle  environments  in  which  many  objects  move  independently).  Present  methods  of 
3D  display,  with  hidden  parts  removed,  are  generally  time  consuming  and  applicable  only  to  the  display  of 
simple  scenes  in  a man-computer  symbiotic  system.  Promising  areas  of  research  may  lie  in  further  variation 
of  sorting  orders  (with  respect  to  the  x,y,  and  z axes)  and  in  making  the  sorting  order  dependent  on  the 
image  space  environment  rather  than  fixing  it  at  the  onset. 

Language.  Work  on  developing  an  efficient  language  for  graphics  applications  has  included  *>■' 
development  of  programming  languages  and  small-scale  voice  input  techniques,  with  the  former  being 
several  orders  of  magnitude  more  prevalent. 

( 1 ) Graphic  Programming  languages 

There  is  nearly  a one  to  one  mapping  from  the  set  of  graphic  programming  languages  onto  the 
(sizeable)  set  of  interactive  graphics  applications.  A survey  of  the  entire  area  would  have  easily  resulted  in 
an  additional  paper  nearly  equal  in  length  to  this  one.  Therefore,  wfiat  follows  is  more  a general  impression 
than  a survey,  the  latter  being  necessarily  deferred  as  a possible  sequel  to  the  effort  being  reported. 

Graphic  programming  languages  seem  to  be  classifiable  on  the  .basis  of  the  strength  of  their 
association  with  FORTRAN  or  some  other  high  level  language.  The  number  of  languages  per  class  is  greatest 
for  those  classes  having  strongest  FORTRAN  associations,  dwindling  rapidly  as  that  strength  lessens  and 
finally  disappears.  This  is  undoubtedly  due  to  the  (commendable)  goal  of  producing  a universal  graphic 
language,  which  is  partly  achievable  by  basing  it  upon  a universal  high  level  language  already  in  existence. 

FORTRAN  subroutine  packages  are  probably  the  most  widely  used  of  graphic  languages  (though 
strictly  speaking  a subroutine  package  cannot  properly  be  considered  a language).  Typical  of  these  are  GSP 
(Rully,  1968),  DISPLAYTRAN  (Gagliano.  Thombs,  & Cornish,  1968),  and  ARTA(Mezei  & Zivian,  1972) 
Use  of  such  packages  simply  involves  preparation  of  a FORTRAN  main  program  which  calls  various 
graphics  subroutines  as  required.  The  subroutines  are,  necessarily,  graphic  device  dependent.  In  addition, 
long  argument  lists  make  use  of  the  subroutines  cumbersome. 

Attempts  to  minimize  requirements  for  frequent  subroutine  calls  with  long  calling  vectors  lie 
primarily  in  the  development  of  FORTRAN  extensions  and  additions  (Bracchi  & Somalvico,  1970;  Bracchi 
& Ferrari,  1971;  Dell’  Aquila,  Murro,  Prezioso,  & Refice,  1972;  Frank,  1968,  Hurwitz,  Citron.  & Yeaton. 
1967,  Shearing,  1971)  This  includes  the  development  of  such  FORTRAN  enhancements  as  the  definition 
of  graphic  variables  and  extension  of  the  language  itself  to  include  such  commands  as  DRAW,  FRASL,  and 
EXPAND.  Considerable  dependence  on  subroutine  calls  still  exists  but  is  lessened  by  such  language 
enhancements.  ; 

Lecarme  (1972)  points  out  several  weaknesses  of  graphic  languages  with  strong  FORTRAN 
associations.  He  says  that  calling  vectors  for  subroutine  calls  o ten  contain  10  and  sometimes  up  to  39  53 
parameters.  Also,  the  position  of  the  parameters  in  the  calling  \pctor  is  significant,  and  awkward  techniques 
are  required  to  omit  parameters  when  desired.  He  also  clains  that  many  subroutines  themselves  are 
anonymous  under  their  “foreshortened  names"  and  difficult  t of  use  if  only  for  that  reason. 

These  observations  have  led  to  the  development  of  a number  of  special  purpose  machine-dependent 
graphics  languages  having  little  or  no  association  with  FORTRAN  and  little  or  no  commonalities  among 
themselves  (Hormann  et  al.,  1971 ; Lecarme,  1972,  Newman,  1973;  Sibley,  Taylor.  & Gordon,  1968).  One 
such  language,  for  example,  is  LOGO  (Newman,  1973),  which  consists  of  a set  of  macros  typified  by 

LINETO  x,y 

DISPLAY  s s is  predefined  subroutine 

DRAW  BOX  IN  x,,y,,Xj ,y2 

Another  is  UALOIormann,  Leal,  & Grande)!.  1971 ),  which  is  list  oriented,  permitting  assignments  resulting 
in  pointer  changing  and  value  changing,  as  well  as  having  its  own  unique  set  of  macros 

The  overriding  impression  gained  from  a cursory  review  of  existing  graphic  programming  languages  is 
that  those  attempting  to  be  general  purpose  arc  far  less  than  optimal,  while  those  which  liave  more 
extensive  flexibility  are  application  dependent  and  would  need  to  be  reprogrammed  for  use  in  a given 
man-computer  symbiotic  system.  Also,  as  with  most  programming  languages,  considerably  more  attention 
has  been  paid  to  the  efficacy  of  the  language  in  the  eyes  of  the  computer  than  in  the  eyes  of  the  man  Thus, 
most  appear  potentially  cumbersome,  slow,  and  unnatural  to  use  in  a system  purporting  to  produce  a 
symbiotic  relationship  between  man  and  machine.  Perhaps  equally  important,  graphic  programming 
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languages  per  sc  seem  highly  unsuited  to  symbiotic  system  use  where  attention  getting  and  rapid  interaction 
may  occasionally  be  required. 

(2)  Small-Scale  Voice  Input  Techniques 

Since  most  computers  are  d^caf,  the  simplest  and  most  common  practice  is  to  treat  man  as  a mute,  as 
in  the  case  of  graphic  programming  languages.  large-scale  (i.e.,  unlimited  vocabulary)  voice  input 
capabilities  are  certainly  a potential  tool  in  man-computer  symbiotic  systems,  but  at  least  for  the  very  near 
future,  the  state  of  the  art  does  not  suggest  their  widespread.  low-c«Kt  availability.  However,  several  fairly 
successful  attempts  have  been  made  to  provide  a limited  voice  input  capability  that  is  well  within  the  state 
of  the  art  and  ideally  suited  to  man-computer  symbiotic  systems. 

Small-scale  voice  input  techniques  perform  four  basic  functions.  First,  a speech  processor  (usually 
analog)  extracts  signals  from  the  input  waveform  that  will  be  used  to  characterize  the  utterance.  Bagley 
(1972)  for  example  extracted  three  measures  (a)  a measure  of  signal  level,  proportional  to  short-term 
average  power;  (b)  a measure  of  zero  crossing  rate  of  the  signal;  and  (c)  a measure  of  zero  crossing  rate  of 
the  first  time  derivative  of  the  signal.  Klder  ( 1970)  used  a group  of  bandpass  filters  to  extract  time-sampled 
measures  prpportional  to  the  power  output  of  each  filter  (similar  to  Fourier  analysis). 

The  second  task  is  to  recognize  when  a word  or  utterance  has  been  spoken.  This  requires  imposition 
of  a restriction  that  a moment  of  silence  occur  between  each  word  so  that  word  boundaries  may  be 
detected  (Elder,  1970).  Then  the  recognition  algorithm,  testing  speech  samples  continuously,  scans  the 
amplitude  signal  and  waits  for  it  to  rise  above,  then  fall  below,  some  fixed  threshold  to  detect  the  utterance 
boundaries. 

The  third  task  is  to  model  an  utterance  when  its  occurrence  has  been  recognized.  Elder’s  technique  is 
to  store  an  Ajj  array  where  each  entry  is  proportional  to  the  power  output  of  filter  i at  time  instant  j. 
Bagley  forms  a 22-dimensional  characteristic  vector  by  breaking  each  utterance  into  7 equal  length  blocks 
and  averaging  each  of  the  3 measures  witliin  each  block.  A 22nd  element  is  computed  which  is  proportional 
to  the  duration  of  the  utterance. 

The  fourth  task  is  to  decide  which,  of  a limited  number  of  utterances,  has  been  spoken.  This  decision 
task  would  fx-  formidable  were  it  not  for  the  imposition  of  restrictions  on  vocabulary  size  and  training  the 
system  to  recognize  only  known  speakers.  These  are  the  two  major  restrictions  which  distinguish  small-scale 
voice  input  capabilities  from  their  general  purpose,  large-scale  counterparts. 

(a)  Training  the  System 

The  basic  task  in  training  the  system  is  to  first  compile  reference  models  for  each  utterance.  In 
application,  each  reference  model  is  compared  with  the  current  utterance  model  and  the  decision  task 
becomes  one  of  determining  the  ‘closest  fit.’ 

System  training  is  the  compilation  of  the  reference  models.  One  technique  is  to  compute  a separate 
set  of  reference  models  for  each  speaker  and  store  them  in  some  form  of  mass  storage.  This  is  the  method 
used  by  Elder  (1970).  Then,  when  a previous  user  approaches  the  system,  his  reference  anays  are  fetched 
for  use  in  the  decision  task.  Training  is  performed  by  asking  each  new  user  to  recite  the  vocabulary  set 
several  times,  and  computing  a single  model  of  each  utterance  by  using  an  averaging  method. 

Another  method,  used  by  Bagley  (1972),  is  to  compute  a single  set  of  reference  models  for  all 
speakers.  Tins  is  accomplished  by  recursively  building  a prototype  vector  dictionary  consisting  of 
characteristic  vectors  (utterance  models)  for  each  utterance  obtained  from  various  speakers.  During 
training,  this  prototype  vector  dictionary  is  used  in  a closest-match  scheme  to  guess  which  utterance  has 
been  s|xiken.  II  the  guess  is  correct,  no  updating  is  performed  on  the  dictionary.  If  incorrect,  the 
characteristic  vector  for  the  utterance  is  added  to  the  dictionary  and  testing/training  continues.  This  is 
continued  up  to  some  preset  limit  on  the  dictionary  size. 

(b)  The  Decision  Process 

Deciding  which  utterance  has  been  spoken  requires  the  computation  of  a measure  of  closeness 
between  the  model  or  characteristic  vector  for  the  input  and  the  reference  models  or  vectors.  The  typical 
approach  is  to  compute  a summed-square  difference  error  over  the  respective  components  of  the  two 
vectors  being  compared  (Bagley,  1972,  Elder,  1970).  Other  possibilities  are  to  form  a scalar  or  dot  product 


or  an  absolute  difference  error.  A disadvantage  of  this  technique  is  that  it  requires  computation  of  the 
selected  measure  for  every  reference  model  vector. 

Elder  ( I ‘>70)  proposes  (but  did  not  implement)  a technique  to  reduce  this  computational  load  by 
linutmg  the  number  of  reference  models  that  are  compared  with  die  model  for  the  unknown  utterance,  lie 
suggests  that  a prominent-feature  comparison  be  performed  first  to  eliminate  reference  vectors  that  differ 
markedly  from  the  vector  under  test.  Such  features  might  include  t)ie  real-time  length  of  die  utterance  or 
the  number  of  peaks  occurring  in  the  power-versus-tnne  curve  for  an  utterance. 

(c)  Experimental  Results 

Elder’s  technique  (unique  reference  models  for  each  speaker)  resulted  in  a 3.67"  error  in  attempting  to  < 

correctly  identify  words  from  a vocabulary  size  of  37.  The  test  included  3 utterances  of  each  word  in  the 
vocabulary.  In  an  attempt  to  reduce  error,  a modified  decision  scheme  was  contrived  wherein  the  two 
reference  arrays  closest  to  the  unknown’s  array  were  determined.  If  these  two  arrays  were  very  similar  (as 
determined  by  their  summed  square  difference  error),  then  the  utterance  was  “rejected,"  and  no  attempt 
was  made  to  identify  it.  Hie  theory  was  that  in  this  situation,  a slight  perturbation  in  the  manner  of 
speaking  could  have  resulted  in  a different  identification.  With  this  scheme,  errors  were  nonexistent  but 
36%  of  the  utterances  were  rejected.  It  was  concluded  that  the  modified  scheme  has  little  practical  utility 

Experiments  were  also  run  by  Elder  to  determine  the  effect  on  error  frequency  of  reducing  the 
number  of  filters  used  to  obtain  basic  model  variables  for  the  utterances.  In  the  original  configuration,  16 
filters  were  used,  with  crossover  frequencies  ranging  from  100  to  4,100  Hz.  Reducing  the  number  of  filters 
to  8,  while  retaining  the  same  bandwidth,  resulted  in  an  increase  of  errors  from  3. 67  to  6.3%. 

Bagiev’s  technique  (single  set  of  reference  models  for  all  speakers)  resulted  in  error  frequencies  which 
depended  on  the  number  of  utterances  made  by  each  speaker  during  training.  When  applied  to  a speaker 
who  had  been  involved  in  the  training  process,  errors  ranged  from  about  10%  for  3 repetitions  of  each  word 
in  training,  to  2%  for  30  repetitions.  The  vocabulary  size  was  33.  When  applied  to  a speaker  who  had  not 
been  involved  in  the  training  process,  the  error  frequency  was  14%.  Tins  figure  was  reduced  to  2%  by 
selectively  dropping  from  the  vocabulary  8 words  which  had  the  highest  associated  error  rates. 

Both  methods  were  implemented  on  an  IBM  360/65  computer.  Response  time  for  the  Elder  method 
was  about  185  Ms,  and  that  for  the  Bagley  method  was  215  to  240  Ms.  For  core  storage  of  the  reference 
models,  Elder’s  method  required  9,472  bytes  for  the  37  word  vocabulary,  and  Bagley 's  method  required 
4,400  bytes  for  the  33  word  vocabulary. 

(d)  Observations 

It  appears  that  a distinct  time/meniory  tradeoff  exists  between  the  techniques  of  modeling  each 
speaker  independently  (Elder,  1970)  and  compiling  an  overall  model  for  all  speakers  (Bagley.  P>72).  The 
former  is  faster  but  consumes  about  twice  the  core  for  storing  the  reference  model.  With  adequate  training. 

Bagley's  method  achieves  a slight  increase  in  accuracy  over  Elder’s,  but  this  observation  must  be  weighed  in 
light  of  the  differing  vocabulary  sizes  used  in  the  two  experiments.  A distinct  advantage  of  Bagley's  method 
is  its  usability  with  unknown  speakers.  This  can  be  performed  with  an  accuracy  equal  to  that  achieved  for 
known  speakers  if  the  vocabulary  set  can  be  judiciously  reduced  to  eliminate  the  most  troublesome  words. 

(3)  Review  and  Conclusions 

A review  of  the  language  aspect  of  computer  graphics  was  inspired  by  the  symbiotic  system 
requirement  for  an  unrestrictive  and  easily  applied  means  of  "getting  the  point  across"  in  transmitting 
information  that  is  relevant  to  system  performance.  Along  with  this  wxs  the  companion  requirement  tor 
some  attention-getting  capability. 

Graphic  programming  languages  are  abundant  and  varied,  but  the  most  prevalent  are  those  which  have 
strong  associations  with  FORTRAN  or  some  other  high  level  language  These  include  FORTRAN 
subroutine  packages  as  well  as  various  additions  to  and  extensions  of  I ORIRAN.  Partly  in  an  attempt  to 
avoid  the  awkwardness  of  using  many  subroutine  calls  with  lengthy  calling  vectors,  efforts  have  also  been 
devoted  to  developing  special  purpose  languages  that  do  not  depend  on  an  association  with  a higher  level 
language.  These  types  of  languages  are  largely  application  specific  and  highly  machine  dependent 

Graphics  programming  languages  are  geared  primarily  to  adaptation  of  the  man  to  the  computer  and, 
in  their  use,  man  is  denied  the  advantage  of  his  most  natural  communication  mode  speech.  However,  a few 
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promising  techniques  exist  which  may  be  immediately  applied  in  man-computer  symbiotic  systems  for 
voice  input  capabilities.  On  a small  scale,  limited  vocabulary  voice  input  techniques  liave  been 
experimentally  developed  with  as  little  as  2%  error  in  recognition  accuracy.  The  principal  variants  of  these 
techniques  are  the  method  of  modeling  the  input  waveform  and  the  method  of  developing  a reference  set  of 
models  (training  the  system)  Response  time  and  memory  tradeoffs  exist  largely  as  a function  of  whether 
each  known  speaker  has  his  own  reference  set  of  models,  or  a single  model  exists  for  all  known  speakers 
Accuracy  is  not  significantly  affected  by  these  alternatives  if  sufficient  time  and  trials  are  devoted  to 
training  the  system. 

In  conclusion,  graphic  programming  languages  are  probably  adequate  for  communications  that  (a) 
are  anticipated  b\  the  transmitter,  and  (b)  do  not  require  speed.' However,  as  with  nearly  any  programming 
language,  they  are  at  best  a communication  aid  that  must  be  learned  and  which  necessitates  planning  and 
time  to  apply  I or  this  reason,  they  do  not.  by  themselves,  fulfill  all  language  requirements  for  a symbiotic 
system,  where  both  attention  getting  and  extensive  flexibility  may  be  required  to  transmit  information 
critical  to  on-going  system  performance. 

An  extremely  promising  adjunct  to  a symbiotic  system  is  a small-scale  voice  input  capability.  This 
would  provide  a highly  efficient  attention-getting  capability  for  one  tiling,  and  possibly  could  be  used  in 
conjunction  with  a graphic  programming  language  to  provide  an  effective  overall  means  of  communication. 
Hill  (l‘)7l)  cites  several  other  advantages  of  such  a capability,  including  additional  reliability  and  its 
naturalness  to  the  man.  Equally  important  is  the  potentially  favorable  effect  that  it  could  have  on  the  man’s 
attitude  about  the  system  and  his  rapport  with  the  computer.  A small  vocabulary  si/e  does  not  appear  to  be 
a serious  limitation  if  the  vocabulary  were  judiciously  selected.  However  error  rate  may  be  critical,  because 
use  of  voice  input  would  have  its  greatest  utility  in  situations  where  speed  and  accuracy  are  both  relevant, 
(l  or  instance,  mistaking  “No"  for  “GO”  at  a critical  point  in  system  operation  could  be  disastrous.)  The 
most  fruitful  areas  of  research  in  the  overall  language  area  seem  to  be:  (a)  reducing  error  rale  in  small-scale 
voice  input  techniques,  and  (b)  combining  graphic  programming  languages  and  voice  input  techniques  into  a 
single  efficient  communication  capability. 


IV.  OVERVIEW  AND  CONCLUSIONS 

The  purpose  of  this  paper  was  lo  determine  and  survey  the  research  areas  that  appear  most  critical  to 
achieving  man-computer  symbiosis,  l irst,  an  operational  definition  of  man-computer  symbiosis  wxs  denved 
which  distinguishes  it  from  other  types  of  man-computer  relationships.  Tins  was  accomplished  by  viewing 
such  relationships  from  a systems  standpoint  and  characterizing  them  by  the  types  of  information  or  data 
that  are  transccived  by  man  and  computer.  The  primary  distinguishing  feature  of  a symbiotic  system  is  the 
capability  of  both  parties  to  transccivc  relevant  unrequested  information. 

Using  the  derived  definition,  key  system  requirements  were  projected.  These  included  status 
transceivers  and  memories,  graphic  techniques,  a language,  and  an  attention-getting  capability.  A survey  was 
then  performed  of  research  and  the  state  of  the  art  of  technology  addressing  each  of  these  requirements. 
Critical  problems  were  identified  during  the  course  of  the  survey,  and  emphasis  was  placed  on  a review  of 
research  areas  related  thereto. 

Status  Transceivers  and  Memories 

It  was  concluded  that  raster  scan  CRTs  hold  the  greatest  promise  for  symbiotic  system  applications 
due  mainly  to  the  reduced  refresh  rates  required  when  using  an  interlaced  scan.  This  provides  a capability 
for  transmitting  larger  quantities  of  data  without  incurring  flicker  in  the  display,  which  is  a major  source  of 
trouble  in  many  graplucs  applications.  While  raster  scan  CRT  technology  itself  [loses  no  specific  research 
problems,  requirements  for  efficient  scan  conversion  methods  do. 

Input  devices  arc  many  and  varied,  but  there  is  no  single  device  that  is  low  cost  and  optimal  for 
pointing,  drawing,  and  positioning  data  on  the  CRT.  These  are  relatively  significant  capabilities  to  develop 
for  man-computer  symbiosis  because  pointing,  drawing,  and  positioning  arc  the  major  actions  by  which 
man  can  communicate  with  the  computer  using  the  display.  The  light  pen  is  a relatively  flexible  input 
device,  but  taster  response  times  are  needed.  It  is  also  desirable,  from  the  standpoint  of  convenience  to  the 
human,  to  provide  a stylus-independent  input  device.  However,  the  poor  quality  of  achievable  resolution  is 
a major  problem  here. 


Respite  a lot  of  evidence  as  to  its  n^ed,  little  work  has  been  accomplished  in  efficiently  extending 
man's  memory  precision  and  recall  capability.  Hie  most  promising  approach  seems  to  be  to  provide  him  a 
simulated  short-term  memory.  Only  one  study  was  found  which  attempted  to  do  this  in  an  interactive 
graphics  application.  It  was  application  specific,  however,  and  not  directly  generali/able  toman-computer 
symbiotic  systems.  Efforts  to  extend  computer  memory  capacity  have  mainly  consisted  of  improved  data 
structures  to  enhance  retrieval  speed.  There  is  some  evidence  that  perhaps  a better  direction  of  effort  would 
be  to  reduce  the  number  of  display  files  typically  required. 

(iraphics  Techniques 

The  major  research  problems  existing  in  the  area  of  graphics  techniques  are  the  development  of 
efficient  scan  conversion  methods  and  techniques  for  solving  the  hidden  line/surface  problem.  Existing  scan 
conversion  methods  are  either  too  time  consuming  or  too  memory  consuming.  Attempts  to  improve  on  one 
of  these  aspects  have  resulted  in  tradeoffs  on  the  other.  Hardware  scan  conversion  is  possible,  but  only  at  a 
high  cost  and  with  highly  complex  electronic  equipment  which  detracts  from  overall  system  reliability.  The 
only  real-time  solution  to  the  hidden  line/surface  problem  requires  enormous  processing  power  for  a 
large-scale  implementation,  and  is  not  suitable  for  use  with  environments  where  everything  moves 
independently.  Other  solutions  are  extremely  slow  in  general,  and,  again,  speed/memory  tradeoffs  have 
occurred  in  every  attempt  at  improvement. 

Tracking  the  light  pen  continues  to  be  a problem  due  to  the  possibility  of  the  tracking  program 
getting  lost  if  the  pen  is  moved  too  rapidly.  Since  the  light  pen  is  one  of  the  lowest  cost  input  devices 
available  and  is  fairly  flexible,  its  use  in  near-future  symbiotic  systems  is  anticipated.  Therefore,  pen 
tracking  may  be  considered  one  research  problem  relevant  to  symbiotic  system  development. 

Even  though  raster  scan  CRTs  hold  greater  promise  than  do  random  scan  types  due  to  the  flicker 
problem,  they  do  have  the  companion  requirement  for  scan  conversion.  One  way  of  reducing  the  flicker 
probability  in  random  scan  vector  drawing  systems  is  to  reduce  the  number  of  vectors  needed  to  compose  a 
scene.  Visual  resolution  processing  seems  to  be  one  technique  for  accomplishing  this.  It  certainly  merits 
further  investigation  to  determine  whether  or  not  its  full-scale  use  in  conjunction  with  random  scan  systems 
can  produce  flicker-free  displays  whose  densities  approach  those  possible  with  raster  scan  systems. 

Language 

The  most  promising  research  area  here  is  the  further  development  and  refinement  of  small-scale  voice 
input  techniques.  Improved  accuracy  seems  to  be  the  most  significant  requirement,  litis  would  not  only 
provide  a highly  efficient  attention-getting  capability,  but  w ould  enhance  the  rapport  which  man  feels  with 
the  computer  by  allowing  him  to  use  his  most  natural  communication  mode.  Another  promising  area  is  to 
investigate  hybrid  communication  techniques  using  a combination  of  graphic  languages  and  small-scale 
voice  input  methods.  Tins  could  offset  some  of  the  disadvantages  of  graphic  languages  (cumbersome,  slow 
and  require  training  the  person)  by  complementing  their  use  with  the  speed  and  naturalness  of  speech. 

Summary  of  Critical  Research  Areas 

Table  3 provides  a summary  of  the  research  areas  determined  to  be  most  critical.  Advancements  in 
these  areas  would  effect  the  most  rapid  closure  on  achieving  the  type  of  system  required  for  man-computer 
symbiosis.  The  interesting  and,  in  a way,  discouraging  observation  that  evolves  from  Table  3 is  that  many  of 
these  research  areas  arc  not  new  and,  to  one  extent  or  another,  have  already  been  investigated  for  about  a 
decade.  In  view  of  this,  one  wonders  how  much  can  be  expected  in  way  of  further  improvements  in  the 
next  decade. 

On  the  more  optimistic  side,  many  of  the  areas  have  enjoyed  some  very  recent  advancements  (past  2 
to  4 years)  which,  if  not  optimal  themselves,  have  suggested  new  avenues  of  investigation.  This  is  especially 
true  of  the  areas  of  scan  conversion,  hidden  linc/surfacc,  and  voice  input.  Still  other  areas  (simulated  STM, 
visual  resolution  processing,  and  the  idea  of  hybrid  communication  techniques)  are,  as  yet,  relatively 
unexplored  and  therefore  wide  open  as  far  as  potentially  bndging  the  gap  to  man -computer  symbiosis, 

This  paper  has  determined  and  reviewed  what  appear  to  be  the  most  fruitful  areas  in  which  we  can 
channel  our  next  efforts.  Hopefully,  therefore,  it  has  contributed  to  the  goals  of  man-computer  symbiosis 
by  serving  as  a point  of  departure  and  possible  inspiration  to  existing  or  potential  aspirants  to  those  goals. 


Table  3.  Summary  of  Critical  Research  Areas* 


Status  Transceivers 
and  Memories 

Graphic  Techniques 

Languaqe 

At  tent  Ion -Get  tin? 
Capability 

1.  Raster  scan  CRTs  & 
associated  software 

l.» Scan  conversion 

1.  Small-scale  voice 
input  techniques 
(improved  accuracy) 

1.  Small-scale 
voice  input 
techniques 

2.  Simulated  STM 

2.  Hidden  line/ 
Surface  techniques 

• 

2.  Hybnd  communication 
methods  (voice  input 
+ graphic  language) 

3.  Low  cost,  general 
purpose  input 
devices 

a.  Sorting  Order 

b.  Variable  sorting 
order 

a.  Liglrt  pen  response 
time 

c.  Image  space 
techniques 

b.  Stylus-independent 
devices  (resolution) 

3.  Visual  resolution 
processing  with 
random  scan  CRTs 

4.  Reduction  of  number 
of  display  files 

4.  Pen  tracking 

aIn  each  category,  research  areas  are  listed  in  descending  order  (author's  opinion)  of  criticality. 
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